Flatten article storage: one file per article, shared author media folder (AI)
Articles no longer live in their own per-post directory — each is now a single `<slug>~<DDMMYYYY>~<tag1>+<tag2>.md` file directly inside its author's directory, alongside that author's bio file and a shared `media/` folder for images. This lets multiple articles reference the same picture without duplicating it per article. discovery.js: buildStub/resolveStub parse the article's own file name instead of a wrapping directory's; indexing's per-author listing now filters to `.md` files (excluding the bio file) rather than subdirectories, and the author's own media/ folder is simply not a `.md` file, so it's excluded for free. The article file's path is now also its own cache key (the `root` concept is gone). cache.js: bumped CACHE_SCHEMA_VERSION (cache key shape changed) and renamed the object store from the now-inaccurate "slugDirs". Migrated all existing test content (10,200 articles) to the new flat layout; no code changes needed to the articles' own content. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXSVPS3UmmASiiprPfcnUd
This commit is contained in:
1 parent
5c716a3f9b
commit
997768e2ae
4 files changed
+148
-128
No files matched your search
@@ -4,18 +4,20 @@ A minimal CMS where articles are Markdown files rendered in the browser. No buil
|
||||
|
||||
## How it works
|
||||
|
||||
1. The browser traverses `cmjs/articles/` via a server directory index — discovering authors and their post directories; nginx JSON autoindex is preferred, with HTML autoindexes supported as a fallback.
|
||||
2. Sorting, filtering, and publish status all come from the post directory's own *name*
|
||||
(`slug~DDMMYYYY~tags`) — knowable straight from the listing that discovers it, with no
|
||||
1. The browser traverses `cmjs/articles/` via a server directory index — discovering authors and
|
||||
their article files; nginx JSON autoindex is preferred, with HTML autoindexes supported as a
|
||||
fallback.
|
||||
2. Sorting, filtering, and publish status all come from the article file's own *name*
|
||||
(`slug~DDMMYYYY~tags.md`) — knowable straight from the listing that discovers it, with no
|
||||
further request per article. A `.md` file's actual content (title, description, image) is
|
||||
only fetched once that article is about to be shown — the current page's cards, a single
|
||||
article, or a search — not for the whole site up front. See "Adding an article" and "Article
|
||||
discovery & caching" below.
|
||||
3. Clicking an article fetches the Markdown file and renders it with [marked.js](https://marked.js.org/).
|
||||
4. Hash-based routing handles navigation — back/forward works without page reloads.
|
||||
5. Once an article's content has been fetched, it's cached in IndexedDB, keyed by post directory
|
||||
and invalidated by directory `mtime`, so repeat visits skip re-fetching unchanged content —
|
||||
see "Article discovery & caching" below.
|
||||
5. Once an article's content has been fetched, it's cached in IndexedDB, keyed by the article
|
||||
file's own path and invalidated by that file's `mtime`, so repeat visits skip re-fetching
|
||||
unchanged content — see "Article discovery & caching" below.
|
||||
|
||||
## Features
|
||||
|
||||
@@ -58,7 +60,7 @@ cmjs/
|
||||
│ ├── frontmatter.js # parseFrontmatter() — pure YAML-front parser
|
||||
│ ├── templates.js # loadTemplates(), templates cache
|
||||
│ ├── discovery.js # discoverArticles(), fetchJson(), article cache
|
||||
│ ├── cache.js # IndexedDB cache for discovered articles, keyed by post directory
|
||||
│ ├── cache.js # IndexedDB cache for discovered articles, keyed by article file path
|
||||
│ ├── concurrency.js # mapWithConcurrency() — caps simultaneous per-article fetches
|
||||
│ ├── i18n.js # t(), loadStrings(), getLocale() — UI string lookup
|
||||
│ ├── views.js # loadArticleList, loadArticle, loadAuthorView, loadCategoryView, loadSearch
|
||||
@@ -76,24 +78,25 @@ cmjs/
|
||||
└── articles/
|
||||
└── author/ # One directory per author
|
||||
├── author.md # Author profile (optional)
|
||||
├── avatar.jpg # Optional — referenced by author.md's `avatar:` field
|
||||
└── slug~DDMMYYYY~tag1+tag2/ # One directory per post — see "Adding an article"
|
||||
├── slug.md # Article content + frontmatter
|
||||
└── cover.jpg # Optional — referenced by slug.md's `image:` field
|
||||
├── media/ # Shared images for this author — see "Adding an article"
|
||||
│ ├── avatar.jpg # Referenced by author.md's `avatar:` field
|
||||
│ └── cover.jpg # Referenced by one or more articles' `image:` field
|
||||
└── slug~DDMMYYYY~tag1+tag2.md # One file per article — see "Adding an article"
|
||||
```
|
||||
|
||||
## Adding an article
|
||||
|
||||
An article's directory *name* is where its publish date, tags, category, and draft status all
|
||||
live — not in separate files, and not in frontmatter. The format is:
|
||||
An article is a single Markdown file placed directly in its author's directory. Its *file name*
|
||||
is where its publish date, tags, category, and draft status all live — not in separate files, and
|
||||
not in frontmatter. The format is:
|
||||
|
||||
```
|
||||
<slug>~<DDMMYYYY>~<tag1>+<tag2>+…
|
||||
<slug>~<DDMMYYYY>~<tag1>+<tag2>+….md
|
||||
```
|
||||
|
||||
- `<slug>` — the stable identifier used in the article's URL (`#post/author/slug`). Treat this
|
||||
as permanent: retagging or correcting the date later just means renaming the directory (see
|
||||
below), but changing the slug changes the article's URL.
|
||||
as permanent: retagging or correcting the date later just means renaming the file (see below),
|
||||
but changing the slug changes the article's URL.
|
||||
- `<DDMMYYYY>` — the publish date, 8 digits, no separators (e.g. `16062026` for 16 June 2026).
|
||||
- `<tag1>+<tag2>+…` — zero or more tag ids, joined with `+`. A tag id matching `expa-YYYY` is
|
||||
treated as the article's "year"/category, shown as a badge linking to `#category/expa-YYYY`;
|
||||
@@ -104,32 +107,35 @@ The separator is `~` (tilde), not a hyphen — deliberately, since a slug is exp
|
||||
kebab-case (single hyphens only) and `~` isn't something you'd type by accident while writing
|
||||
one, unlike an extra `-`.
|
||||
|
||||
A leading underscore on the whole directory name — `_<slug>~...` — marks the article as a
|
||||
A leading underscore on the whole file name — `_<slug>~....md` — marks the article as a
|
||||
**draft**: it's excluded from the list, search, and pagination, but still reachable directly by
|
||||
URL. Remove the underscore to publish it.
|
||||
|
||||
Images (a cover photo, inline pictures) live in a `media/` folder shared by all of that author's
|
||||
articles, not next to each individual article file — so the same picture can be reused across
|
||||
several articles without keeping a duplicate copy per article. Reference one from frontmatter or
|
||||
markdown with a path relative to the author directory, e.g. `image: media/photo.jpg`.
|
||||
|
||||
Steps:
|
||||
|
||||
1. Create the author directory if it doesn't exist: `cmjs/articles/author/`
|
||||
2. Create the post directory using the naming convention above, e.g.
|
||||
`cmjs/articles/author/my-article~16062026~kometa+expa-2026/`
|
||||
3. Create the Markdown file inside, named after the slug: `my-article.md`
|
||||
4. Optionally add a cover image beside it too, e.g. `photo.jpg`, and reference it from
|
||||
frontmatter with `image: photo.jpg`. Images live next to the article that uses them, not in a
|
||||
shared folder.
|
||||
5. Add a frontmatter block at the top of the Markdown file
|
||||
6. Write your content — it will appear in the list automatically
|
||||
2. Create the Markdown file directly inside it, using the naming convention above, e.g.
|
||||
`cmjs/articles/author/my-article~16062026~kometa+expa-2026.md`
|
||||
3. Optionally add a cover image to that author's shared `cmjs/articles/author/media/` folder
|
||||
(creating it if needed), and reference it from frontmatter with `image: media/photo.jpg`.
|
||||
4. Add a frontmatter block at the top of the Markdown file
|
||||
5. Write your content — it will appear in the list automatically
|
||||
|
||||
To change an article's date, tags, or draft status later, rename its directory — the article's
|
||||
`slug.md` filename and content don't need to change. `author` is derived from the directory path
|
||||
— no frontmatter field needed for it either. Because every one of these is encoded in the
|
||||
directory name, discovering an article's metadata never requires fetching anything beyond the
|
||||
directory listing that already reveals it — no further per-article request at all, which is what
|
||||
makes the "Article discovery & caching" section below possible.
|
||||
To change an article's date, tags, or draft status later, rename its file — its content doesn't
|
||||
need to change. `author` is derived from the containing directory — no frontmatter field needed
|
||||
for it either. Because every one of these is encoded in the file name, discovering an article's
|
||||
metadata never requires fetching anything beyond the directory listing that already reveals it —
|
||||
no further per-article request at all, which is what makes the "Article discovery & caching"
|
||||
section below possible.
|
||||
|
||||
### Tag labels
|
||||
|
||||
A tag in the directory name is just an id (e.g. `kometa`) — the human-readable label shown on
|
||||
A tag in the file name is just an id (e.g. `kometa`) — the human-readable label shown on
|
||||
tag pills comes from `cmjs/blog-config/tags.json`:
|
||||
|
||||
```json
|
||||
@@ -160,7 +166,7 @@ Your markdown content here...
|
||||
|-------|----------|-------|
|
||||
| `title` | yes | Shown in the list and article header |
|
||||
| `description` | no | Short summary shown under the title in the list |
|
||||
| `image` | no | Filename of an image file placed in this same post directory (e.g. `photo.jpg`); shown as card thumbnail and article hero |
|
||||
| `image` | no | Path to an image in the author's shared `media/` folder (e.g. `media/photo.jpg`); shown as card thumbnail and article hero |
|
||||
|
||||
Reading time is computed automatically from the article word count — no frontmatter field
|
||||
needed. `date`, `tags`, and `draft` frontmatter fields, if present, are ignored — superseded by
|
||||
@@ -195,7 +201,7 @@ Short bio rendered as markdown.
|
||||
| Field | Notes |
|
||||
|-------|-------|
|
||||
| `title` | Display name shown in the profile card. Falls back to the author directory name. |
|
||||
| `avatar` | Filename of an image file placed in this same author directory — displayed as a circular thumbnail. |
|
||||
| `avatar` | Path to an image, relative to the author directory — e.g. `media/avatar.jpg` if kept in the shared media folder alongside article images, or a bare filename if kept directly beside `{author}.md`. Displayed as a circular thumbnail. |
|
||||
| `github` / `twitter` / `linkedin` / `mastodon` / `website` | Any subset of these renders as pill links below the bio. |
|
||||
|
||||
The body text below the frontmatter is rendered as markdown and shown as the bio.
|
||||
@@ -214,7 +220,7 @@ The body text below the frontmatter is rendered as markdown and shown as the bio
|
||||
| `#category/expa-2026/page/2` | Same, page 2 |
|
||||
| `#tag/kometa` | All articles tagged "kometa", page 1 |
|
||||
| `#tag/kometa/page/2` | Same, page 2 |
|
||||
| `#post/alice/my-article` | Single article at `cmjs/articles/alice/my-article/my-article.md` |
|
||||
| `#post/alice/my-article` | Single article at `cmjs/articles/alice/my-article~DDMMYYYY~tags.md` |
|
||||
|
||||
Author names and category tags on each card and article are clickable links to their respective filter views. A search box in the header navigates to `#search/query`.
|
||||
|
||||
@@ -390,8 +396,10 @@ not built into stock nginx — only worth it if you're already building custom n
|
||||
## Article discovery & caching
|
||||
|
||||
Discovery happens in two phases. **Indexing** crawls `cmjs/articles/` client-side to build a
|
||||
lightweight "stub" for every article — the author directories, then each author's post
|
||||
directories along with their `mtime`. That's it: because a post directory's *name* already
|
||||
lightweight "stub" for every article — the author directories, then each author's own directory
|
||||
listing (their article files, bio file, and shared `media/` folder together) along with each
|
||||
file's `mtime`. Non-article entries (the author's own `{author}.md`, the `media/` folder) are
|
||||
simply filtered out of that same listing. That's it: because an article file's *name* already
|
||||
encodes its slug, date, tags, and draft status (see "Adding an article" above), that single
|
||||
listing is enough to sort, filter, paginate, and populate the sidebar's tag/year counts, without
|
||||
any further request per article and without ever fetching an article's actual Markdown.
|
||||
@@ -403,20 +411,20 @@ viewed actually needs. A large site's list is therefore fast to show regardless
|
||||
articles it has — indexing costs one request per author, full stop, and only ~20 articles' worth
|
||||
of content is ever fetched per page view.
|
||||
|
||||
- **Cache** — hydrated articles are cached per post directory in IndexedDB (`cmjs/js/cache.js`),
|
||||
keyed by directory path and invalidated by comparing the live `mtime` against the cached one —
|
||||
once *any* visit renders an article, every later visit skips its content fetch too, until its
|
||||
directory is renamed (a rename changes the cache key, so a retagged/redated article is treated
|
||||
as new — see "Adding an article" above) or its `.md` file's own mtime changes. Indexing reads
|
||||
the entire cache once up front (`getAllCacheEntries()`, kicked off in parallel with the
|
||||
author/post-directory requests) so a warm cache doesn't cost an IndexedDB read per article. If
|
||||
IndexedDB is unavailable (private browsing, disabled storage), discovery transparently falls
|
||||
back to fetching everything live on every visit.
|
||||
- **Cache** — hydrated articles are cached per article file in IndexedDB (`cmjs/js/cache.js`),
|
||||
keyed by the file's own path and invalidated by comparing its live `mtime` against the cached
|
||||
one — once *any* visit renders an article, every later visit skips its content fetch too, until
|
||||
the file is renamed (a rename changes the cache key, so a retagged/redated article is treated as
|
||||
new — see "Adding an article" above) or its own mtime changes. Indexing reads the entire cache
|
||||
once up front (`getAllCacheEntries()`, kicked off in parallel with the author/directory-listing
|
||||
requests) so a warm cache doesn't cost an IndexedDB read per article. If IndexedDB is unavailable
|
||||
(private browsing, disabled storage), discovery transparently falls back to fetching everything
|
||||
live on every visit.
|
||||
- **Cache schema version** — the cached records store the full derived article shape, not raw
|
||||
content, so a code change to how articles are derived (in `cmjs/js/discovery.js`) can make old
|
||||
cached entries stale in *structure*, not just content — and a directory's `mtime` won't catch
|
||||
that. Bump `CACHE_SCHEMA_VERSION` in `cmjs/js/cache.js` whenever that shape changes; it's baked
|
||||
into the IndexedDB database name, so a bump invalidates every cached entry on the next deploy.
|
||||
cached entries stale in *structure*, not just content — and a file's `mtime` won't catch that.
|
||||
Bump `CACHE_SCHEMA_VERSION` in `cmjs/js/cache.js` whenever that shape changes; it's baked into
|
||||
the IndexedDB database name, so a bump invalidates every cached entry on the next deploy.
|
||||
- **Concurrency** — `MAX_CONCURRENT_REQUESTS` in `cmjs/js/discovery.js` (default `10`) caps how
|
||||
many hydration (content) fetches run at once — the only per-article requests left, since
|
||||
indexing no longer needs one; raise it for a fast/local nginx, lower it for a slow or shared/
|
||||
@@ -435,17 +443,18 @@ of content is ever fetched per page view.
|
||||
page count is exact once the whole set's been scanned, and a lower bound (at least one more
|
||||
page) until then.
|
||||
- **Crawl health** — if every author's directory listing fails outright, or very few of the
|
||||
known post directories parsed successfully (a malformed directory name, or a total breakage
|
||||
like the wrong origin or CORS misconfigured), a small warning banner is shown above the article
|
||||
list instead of silently rendering an empty page (see `getCrawlStats()` /
|
||||
known candidate files parsed successfully (a malformed file name, or a total breakage like the
|
||||
wrong origin or CORS misconfigured), a small warning banner is shown above the article list
|
||||
instead of silently rendering an empty page (see `getCrawlStats()` /
|
||||
`renderCrawlHealthWarning()` in `cmjs/js/discovery.js` / `cmjs/js/views.js`).
|
||||
- **Design rationale** — the original crawler design (wave-based crawling with a per-article
|
||||
listing fetch, recency prioritisation, HTML-autoindex fallback, long-term "deploy and forget"
|
||||
constraints) is documented in
|
||||
[`.claude/nginx-sitemap-crawler-plan.md`](.claude/nginx-sitemap-crawler-plan.md), which ships
|
||||
alongside the code. The per-article listing fetch it describes no longer exists (metadata moved
|
||||
into the directory name instead), but the rest of that reasoning — the caching model, the
|
||||
longevity constraints, the HTML fallback — still applies.
|
||||
into the file name instead, and articles no longer live in a directory of their own), but the
|
||||
rest of that reasoning — the caching model, the longevity constraints, the HTML fallback — still
|
||||
applies.
|
||||
|
||||
## Running locally
|
||||
|
||||
|
||||
+14
-13
@@ -1,18 +1,19 @@
|
||||
// Persistent cache for discovered articles, keyed by slug-directory path.
|
||||
// Raw IndexedDB (no wrapper library — see .claude/nginx-sitemap-crawler-plan.md
|
||||
// L4/L5 on the zero-dependency, long-stable-API requirement). If IndexedDB is
|
||||
// unavailable (private browsing, old browser, blocked), every operation below
|
||||
// silently becomes a no-op so discovery.js gets live-fetch behaviour for free
|
||||
// without needing its own availability branching (plan doc R6).
|
||||
// Persistent cache for discovered articles, keyed by each article file's own
|
||||
// path. Raw IndexedDB (no wrapper library — see
|
||||
// .claude/nginx-sitemap-crawler-plan.md L4/L5 on the zero-dependency,
|
||||
// long-stable-API requirement). If IndexedDB is unavailable (private
|
||||
// browsing, old browser, blocked), every operation below silently becomes a
|
||||
// no-op so discovery.js gets live-fetch behaviour for free without needing
|
||||
// its own availability branching (plan doc R6).
|
||||
|
||||
// Bump this whenever discovery.js's derived article record shape changes.
|
||||
// The version is baked into the DB name so a code deploy invalidates every
|
||||
// cached entry immediately, instead of stale-shaped records lingering until
|
||||
// each directory's mtime happens to change — which, on a "deploy and
|
||||
// forget" site, could be years.
|
||||
const CACHE_SCHEMA_VERSION = 2;
|
||||
// each file's mtime happens to change — which, on a "deploy and forget"
|
||||
// site, could be years.
|
||||
const CACHE_SCHEMA_VERSION = 3;
|
||||
const DB_NAME = `cmjs-cache-v${CACHE_SCHEMA_VERSION}`;
|
||||
const STORE_NAME = 'slugDirs';
|
||||
const STORE_NAME = 'articleFiles';
|
||||
|
||||
let cacheAvailable = true;
|
||||
let dbPromise = null;
|
||||
@@ -58,9 +59,9 @@ export function isCacheAvailable() {
|
||||
|
||||
// Returns every cached { path, mtime, article } record as a Map keyed by
|
||||
// path — one bulk read instead of a separate IndexedDB round-trip per
|
||||
// directory, which matters once there are hundreds of them: discovery.js
|
||||
// reads this once up front rather than doing a point lookup per directory
|
||||
// during indexing. Empty Map on a miss or if the cache is unavailable.
|
||||
// article file, which matters once there are hundreds of them: discovery.js
|
||||
// reads this once up front rather than doing a point lookup per file during
|
||||
// indexing. Empty Map on a miss or if the cache is unavailable.
|
||||
export async function getAllCacheEntries() {
|
||||
const records = await runRequest('readonly', store => store.getAll());
|
||||
return new Map((records || []).map(record => [record.path, record]));
|
||||
|
||||
+70
-60
@@ -7,10 +7,10 @@ import { t } from './i18n.js';
|
||||
// Caps simultaneous markdown content fetches during hydration (see
|
||||
// hydrateArticle()/hydrateArticles()) — the only per-article requests left,
|
||||
// since indexing no longer needs a per-article listing fetch at all (every
|
||||
// stub field is encoded in its directory's own name — see buildStub()). Raise
|
||||
// it for a fast/local nginx, lower it for a slow or shared/production
|
||||
// instance — see .claude/nginx-sitemap-crawler-plan.md §2a for the original
|
||||
// rationale (written when this also governed indexing requests).
|
||||
// stub field is encoded in its file's own name — see buildStub()). Raise it
|
||||
// for a fast/local nginx, lower it for a slow or shared/production instance
|
||||
// — see .claude/nginx-sitemap-crawler-plan.md §2a for the original rationale
|
||||
// (written when this also governed indexing requests).
|
||||
const MAX_CONCURRENT_REQUESTS = 10;
|
||||
|
||||
const MONTH_ABBREVIATIONS = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];
|
||||
@@ -58,11 +58,14 @@ export async function fetchJson(url) {
|
||||
throw new Error(`Expected a directory listing at ${url}`);
|
||||
}
|
||||
|
||||
// Images live alongside whatever references them — an article's cover/inline
|
||||
// images sit in its own slug directory, an author's avatar in their own
|
||||
// author directory — so `articlePath` (the referencing .md file's own URL)
|
||||
// is the resolution base, not a shared folder. A bare filename in
|
||||
// frontmatter therefore always means "next to this file."
|
||||
// Images live alongside whatever references them: every article file sits
|
||||
// directly in its author's own directory (see buildStub()), next to that
|
||||
// author's shared `media/` folder — so `articlePath` (the referencing .md
|
||||
// file's own URL) is the resolution base. A relative path in frontmatter
|
||||
// like `media/cover.jpg` resolves against the article file's own directory,
|
||||
// which is the author directory itself, landing it in that shared folder —
|
||||
// letting multiple articles reference the same image without duplicating it
|
||||
// per article.
|
||||
export function resolveMediaUrl(value, articlePath = '') {
|
||||
if (!value) return '';
|
||||
let path = String(value).trim();
|
||||
@@ -129,8 +132,8 @@ function notifyProgress(done) {
|
||||
progressListeners.forEach(listener => listener(snapshot, done));
|
||||
}
|
||||
|
||||
// How many of the last crawl's slug directories actually produced an article
|
||||
// vs. how many were candidates, plus how many authors' listings failed
|
||||
// How many of the last crawl's candidate files actually produced an article
|
||||
// vs. how many were listed, plus how many authors' listings failed
|
||||
// outright — lets the UI show a "something's badly wrong" indicator instead
|
||||
// of silently rendering an empty/near-empty list forever.
|
||||
export function getCrawlStats() {
|
||||
@@ -149,19 +152,21 @@ function excerpt(content) {
|
||||
return `${text.slice(0, 277).replace(/\s+\S*$/, '')}…`;
|
||||
}
|
||||
|
||||
// A post directory's own name is the sole authoritative source of its
|
||||
// slug, publish date, tags/category, and draft status — see the README's
|
||||
// "Adding an article" section. Format: `<slug>~<DDMMYYYY>[~tag1+tag2+…]`,
|
||||
// An article file's own name is the sole authoritative source of its slug,
|
||||
// publish date, tags/category, and draft status — see the README's "Adding
|
||||
// an article" section. Format: `<slug>~<DDMMYYYY>[~tag1+tag2+…].md`,
|
||||
// optionally prefixed with `_` for draft. `~` (rather than e.g. "--") is
|
||||
// deliberate: a slug is expected to be plain kebab-case, and "~" isn't a
|
||||
// character anyone would type by accident while writing one — unlike a
|
||||
// double-hyphen separator, which collides with an easy slug typo (an extra
|
||||
// "-") and silently drops the whole article instead of erroring. Returns
|
||||
// null for any directory name that doesn't match — callers treat that as a
|
||||
// silent skip, e.g. a stray directory that isn't a post at all.
|
||||
function parseSlugDirName(name) {
|
||||
const draft = name.startsWith('_');
|
||||
const core = draft ? name.slice(1) : name;
|
||||
// null for any file name that doesn't match — callers treat that as a
|
||||
// silent skip, e.g. the author's own bio file or a stray unrelated file.
|
||||
function parseArticleFileName(fileName) {
|
||||
if (!fileName.endsWith('.md')) return null;
|
||||
const base = fileName.slice(0, -3);
|
||||
const draft = base.startsWith('_');
|
||||
const core = draft ? base.slice(1) : base;
|
||||
const [slug, dateSegment, tagsSegment] = core.split('~');
|
||||
if (!slug || !dateSegment || !/^\d{8}$/.test(dateSegment)) return null;
|
||||
const day = dateSegment.slice(0, 2);
|
||||
@@ -176,18 +181,21 @@ function parseSlugDirName(name) {
|
||||
}
|
||||
|
||||
// Everything an article card/filter/sort needs, parsed entirely from its
|
||||
// directory's own name and its listing entry's mtime — no fetch of any
|
||||
// kind. This is what runs for every article at index time; the (expensive)
|
||||
// file's own name and its listing entry's mtime — no fetch of any kind.
|
||||
// This is what runs for every article at index time; the (expensive)
|
||||
// content fetch is deferred to hydrateArticle(), which only ever runs for
|
||||
// articles actually being rendered. Returns null if the directory name
|
||||
// doesn't match the naming convention (see parseSlugDirName) — the caller
|
||||
// articles actually being rendered. Returns null if the file name doesn't
|
||||
// match the naming convention (see parseArticleFileName) — the caller
|
||||
// treats that as a silent skip, matching the rest of this module's
|
||||
// approach.
|
||||
function buildStub(author, dirName, mtime, authorRoot) {
|
||||
const parsed = parseSlugDirName(dirName);
|
||||
// approach. `path` (the article file's own URL) doubles as its cache key —
|
||||
// see cache.js — since every article now lives directly in its author's
|
||||
// directory, alongside that author's bio file and shared `media/` folder,
|
||||
// rather than in a directory of its own.
|
||||
function buildStub(author, fileName, mtime, authorRoot) {
|
||||
const parsed = parseArticleFileName(fileName);
|
||||
if (!parsed) return null;
|
||||
const { slug, date, tags, draft } = parsed;
|
||||
const root = `${authorRoot}${encodeURIComponent(dirName)}/`;
|
||||
const path = `${authorRoot}${encodeURIComponent(fileName)}`;
|
||||
// "Category" isn't a separate concept anymore — it's just whichever tag
|
||||
// id (if any) looks like an expedition year.
|
||||
const category = tags.find(id => /^expa-20\d{2}$/i.test(id)) || null;
|
||||
@@ -196,9 +204,8 @@ function buildStub(author, dirName, mtime, authorRoot) {
|
||||
author,
|
||||
slug,
|
||||
hash: `${author}/${slug}`,
|
||||
root,
|
||||
path,
|
||||
mtime,
|
||||
path: `${root}${encodeURIComponent(slug)}.md`,
|
||||
category,
|
||||
expa: category ? Number(category.replace(/^expa-/i, '')) : null,
|
||||
date,
|
||||
@@ -208,19 +215,19 @@ function buildStub(author, dirName, mtime, authorRoot) {
|
||||
};
|
||||
}
|
||||
|
||||
// Cache-aware: a directory whose mtime matches its cached entry's mtime
|
||||
// reuses that entry directly — worthwhile only when it's already hydrated
|
||||
// Cache-aware: a file whose mtime matches its cached entry's mtime reuses
|
||||
// that entry directly — worthwhile only when it's already hydrated
|
||||
// (skipping a real content re-fetch); a merely-parsed stub is free to
|
||||
// rebuild, so there's no point caching one just to skip re-parsing a
|
||||
// string. `cacheSnapshot` is the whole cache read once up front (see
|
||||
// runCrawl) — an in-memory lookup here instead of a separate IndexedDB read
|
||||
// per directory, which mattered when this also had to guard a listing
|
||||
// fetch; kept anyway since it's still the same one cheap lookup.
|
||||
function resolveStub(author, dirName, mtime, authorRoot, cacheSnapshot) {
|
||||
const root = `${authorRoot}${encodeURIComponent(dirName)}/`;
|
||||
const cached = cacheSnapshot.get(root);
|
||||
// per file, which mattered when this also had to guard a listing fetch;
|
||||
// kept anyway since it's still the same one cheap lookup.
|
||||
function resolveStub(author, fileName, mtime, authorRoot, cacheSnapshot) {
|
||||
const path = `${authorRoot}${encodeURIComponent(fileName)}`;
|
||||
const cached = cacheSnapshot.get(path);
|
||||
if (cached && mtime != null && cached.mtime === mtime && cached.article?.hydrated) return cached.article;
|
||||
return buildStub(author, dirName, mtime, authorRoot);
|
||||
return buildStub(author, fileName, mtime, authorRoot);
|
||||
}
|
||||
|
||||
// Fields only derivable from an article's actual markdown content (not its
|
||||
@@ -261,7 +268,7 @@ export async function hydrateArticle(stub) {
|
||||
// already reflect this same mutation.)
|
||||
const authoritative = { ...stub };
|
||||
Object.assign(stub, meta, authoritative, deriveContentFields(meta, content, stub.path), { hydrated: true });
|
||||
await setCacheEntry(stub.root, { mtime: stub.mtime, article: stub });
|
||||
await setCacheEntry(stub.path, { mtime: stub.mtime, article: stub });
|
||||
} catch { /* leave stub unhydrated — retried next time it's rendered */ }
|
||||
return stub;
|
||||
}
|
||||
@@ -289,49 +296,52 @@ async function runCrawl() {
|
||||
const authors = authorEntries.filter(entry => entry.type === 'directory' && !entry.name.startsWith('.'));
|
||||
|
||||
// Wave 2 — always fetched live, in full, for every author (not just the
|
||||
// current page): each author's post directories, with their mtimes.
|
||||
// This is what builds the flat, cross-author list that drives global
|
||||
// sorting and pagination — see the plan doc's "keep the flat feed"
|
||||
// decision. It's also now the *only* per-directory network request
|
||||
// current page): each author's own directory listing, with per-file
|
||||
// mtimes. This is what builds the flat, cross-author list that drives
|
||||
// global sorting and pagination — see the plan doc's "keep the flat
|
||||
// feed" decision. It's also the *only* per-author network request
|
||||
// indexing ever needs: every stub field (slug, date, tags, category,
|
||||
// draft status) is parsed straight out of each directory's own name
|
||||
// (see buildStub/parseSlugDirName) — there's no more separate
|
||||
// per-article listing fetch the way there used to be. One author's
|
||||
// listing failing (network blip, deleted mid-crawl) is silently
|
||||
// skipped rather than failing every other author's list too, but is
|
||||
// still counted, so a *total* breakage (wrong origin, CORS) remains
|
||||
// visible via getCrawlStats() instead of just silently producing an
|
||||
// empty article list.
|
||||
// draft status) is parsed straight out of each article file's own name
|
||||
// (see buildStub/parseArticleFileName) — there's no per-article fetch
|
||||
// at index time. The author's own bio file (`<author>.md`) and shared
|
||||
// `media/` folder live in the same listing and are simply filtered out
|
||||
// here — only `.md` files other than the bio file are candidate
|
||||
// articles. One author's listing failing (network blip, deleted
|
||||
// mid-crawl) is silently skipped rather than failing every other
|
||||
// author's list too, but is still counted, so a *total* breakage (wrong
|
||||
// origin, CORS) remains visible via getCrawlStats() instead of just
|
||||
// silently producing an empty article list.
|
||||
let authorsFailed = 0;
|
||||
const authorDirLists = await Promise.all(authors.map(async author => {
|
||||
const authorFileLists = await Promise.all(authors.map(async author => {
|
||||
const authorRoot = `${BASE_URL}articles/${encodeURIComponent(author.name)}/`;
|
||||
const bioFileName = `${author.name}.md`;
|
||||
try {
|
||||
const entries = await fetchJson(authorRoot);
|
||||
return entries
|
||||
.filter(entry => entry.type === 'directory' && entry.name !== '_history')
|
||||
.map(entry => ({ author: author.name, dirName: entry.name, mtime: entry.mtime ?? null, authorRoot }));
|
||||
.filter(entry => entry.type === 'file' && entry.name.endsWith('.md') && entry.name !== bioFileName)
|
||||
.map(entry => ({ author: author.name, fileName: entry.name, mtime: entry.mtime ?? null, authorRoot }));
|
||||
} catch {
|
||||
authorsFailed += 1;
|
||||
return [];
|
||||
}
|
||||
}));
|
||||
const allDirs = authorDirLists.flat();
|
||||
const allFiles = authorFileLists.flat();
|
||||
|
||||
// Every stub is parsed synchronously from data already in hand — no
|
||||
// per-directory fetch, so no need for mapWithConcurrency here (that's
|
||||
// still used by hydrateArticles(), which does real network fetches).
|
||||
// per-file fetch, so no need for mapWithConcurrency here (that's still
|
||||
// used by hydrateArticles(), which does real network fetches).
|
||||
const cacheSnapshot = await cacheSnapshotPromise;
|
||||
let loaded = 0;
|
||||
for (const dir of allDirs) {
|
||||
const stub = resolveStub(dir.author, dir.dirName, dir.mtime, dir.authorRoot, cacheSnapshot);
|
||||
for (const file of allFiles) {
|
||||
const stub = resolveStub(file.author, file.fileName, file.mtime, file.authorRoot, cacheSnapshot);
|
||||
if (stub) {
|
||||
loaded += 1;
|
||||
partialResolved.push(stub);
|
||||
}
|
||||
}
|
||||
|
||||
await pruneCache(allDirs.map(dir => `${dir.authorRoot}${encodeURIComponent(dir.dirName)}/`));
|
||||
crawlStats = { total: allDirs.length, loaded, authorsTotal: authors.length, authorsFailed };
|
||||
await pruneCache(allFiles.map(file => `${file.authorRoot}${encodeURIComponent(file.fileName)}`));
|
||||
crawlStats = { total: allFiles.length, loaded, authorsTotal: authors.length, authorsFailed };
|
||||
|
||||
const finalArticles = sortAndFilterPublished(partialResolved);
|
||||
notifyProgress(true);
|
||||
|
||||
+1
-1
@@ -494,7 +494,7 @@ export async function loadArticle(hash) {
|
||||
const element = document.importNode(templates.article, true);
|
||||
const field = name => element.querySelector(`[data-field="${name}"]`);
|
||||
field('title').textContent = meta.title || article.slug;
|
||||
// Publish date comes from the post directory's own name (already
|
||||
// Publish date comes from the article file's own name (already
|
||||
// resolved onto `article.date`), not the raw frontmatter.
|
||||
setField(field('date'), article.date, (time, value) => { time.dateTime = String(value); time.textContent = formatDate(value); });
|
||||
setField(field('description'), meta.description, (description, value) => { description.textContent = value; });
|
||||
|
||||
Reference in new issue
Block a user