Encode article metadata into the post directory name (AI)
Replaces .tag/.date/draft marker files with a single naming convention: <slug>--<DDMMYYYY>--<tag1>+<tag2>+... (optionally prefixed with "_" for draft). This eliminates the per-article listing fetch entirely — every stub field is now parsed straight out of the directory name the author-level listing already returns, so indexing costs one request per author, full stop, regardless of how many articles exist. discovery.js: buildStub()/resolveStub() become synchronous (pure string parsing, no fetch); runCrawl() drops the wave-3 crawl/mapWithConcurrency/mtime-priority-ordering entirely, since there's no more per-item async work to stage. hydrateArticle() and everything downstream in views.js is unchanged, since the derived article shape is identical either way. Removes the example admin/hello-world article (old format, git was tracking it) and regenerates 10 authors x 20 articles in the new format for testing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXSVPS3UmmASiiprPfcnUd
This commit is contained in:
biosiasandClaude Sonnet 5 committed
1 parent
58acdbff98
commit
122593da24
6 files changed
+190
-264
No files matched your search
@@ -4,12 +4,13 @@ A minimal CMS where articles are Markdown files rendered in the browser. No buil
|
||||
|
||||
## How it works
|
||||
|
||||
1. The browser traverses `cmjs/articles/` via a server directory index — discovering authors and their post directories (`author/slug/slug.md`); nginx JSON autoindex is preferred, with HTML autoindexes supported as a fallback.
|
||||
2. Sorting, filtering, and publish status all come from empty marker files beside each article
|
||||
(`.date`, `.tag`, `draft`) — knowable from the directory listing alone. A `.md` file's actual
|
||||
content (title, description, image) is only fetched once that article is about to be shown —
|
||||
the current page's cards, a single article, or a search — not for the whole site up front.
|
||||
See "Adding an article" and "Article discovery & caching" below.
|
||||
1. The browser traverses `cmjs/articles/` via a server directory index — discovering authors and their post directories; nginx JSON autoindex is preferred, with HTML autoindexes supported as a fallback.
|
||||
2. Sorting, filtering, and publish status all come from the post directory's own *name*
|
||||
(`slug--DDMMYYYY--tags`) — knowable straight from the listing that discovers it, with no
|
||||
further request per article. A `.md` file's actual content (title, description, image) is
|
||||
only fetched once that article is about to be shown — the current page's cards, a single
|
||||
article, or a search — not for the whole site up front. See "Adding an article" and "Article
|
||||
discovery & caching" below.
|
||||
3. Clicking an article fetches the Markdown file and renders it with [marked.js](https://marked.js.org/).
|
||||
4. Hash-based routing handles navigation — back/forward works without page reloads.
|
||||
5. Once an article's content has been fetched, it's cached in IndexedDB, keyed by post directory
|
||||
@@ -73,49 +74,59 @@ cmjs/
|
||||
├── blog-config/
|
||||
│ └── tags.json # Tag id → human-readable label registry — see "Tag labels" below
|
||||
└── articles/
|
||||
└── author/ # One directory per author
|
||||
├── author.md # Author profile (optional)
|
||||
├── avatar.jpg # Optional — referenced by author.md's `avatar:` field
|
||||
└── slug/ # One directory per post
|
||||
├── slug.md # Article content + frontmatter
|
||||
├── cover.jpg # Optional — referenced by slug.md's `image:` field
|
||||
├── tag-id.tag # Zero or more empty tag/category markers
|
||||
├── DD-MM-YYYY.date # Publish date marker (exactly one)
|
||||
└── draft # Optional — presence hides the article from the list
|
||||
└── author/ # One directory per author
|
||||
├── author.md # Author profile (optional)
|
||||
├── avatar.jpg # Optional — referenced by author.md's `avatar:` field
|
||||
└── slug--DDMMYYYY--tag1+tag2/ # One directory per post — see "Adding an article"
|
||||
├── slug.md # Article content + frontmatter
|
||||
└── cover.jpg # Optional — referenced by slug.md's `image:` field
|
||||
```
|
||||
|
||||
## Adding an article
|
||||
|
||||
1. Create the author and post directories if they don't exist: `cmjs/articles/author/my-article/`
|
||||
2. Create the Markdown file inside: `cmjs/articles/author/my-article/my-article.md`
|
||||
3. Add a publish-date marker beside it: `cmjs/articles/author/my-article/DD-MM-YYYY.date` (e.g.
|
||||
`16-06-2026.date`) — an empty file, exactly one per article. This is the sole source of the
|
||||
publish date; a `date:` frontmatter field is no longer read.
|
||||
4. Add tag/category markers as needed: `cmjs/articles/author/my-article/tag-id.tag` (e.g.
|
||||
`kometa.tag`, `expa-2026.tag`) — empty files, any number. A tag id matching `expa-YYYY` is
|
||||
treated as the article's "year"/category, shown as a badge on each card/article linking to
|
||||
`#category/expa-YYYY`; every other tag id is a normal topic tag, shown as a pill linking to
|
||||
`#tag/tag-id`.
|
||||
5. Optionally add a cover image beside it too — `cmjs/articles/author/my-article/photo.jpg` —
|
||||
and reference it from frontmatter with `image: photo.jpg`. Images live next to the article
|
||||
that uses them, not in a shared folder.
|
||||
6. To keep it out of the published list for now, add an empty file named exactly
|
||||
`cmjs/articles/author/my-article/draft` (no extension) — remove it (or rename it) when the
|
||||
article is ready to publish. This is the sole source of draft/publish status; a `draft:`
|
||||
frontmatter field is no longer read.
|
||||
7. Add a frontmatter block at the top of the Markdown file
|
||||
8. Write your content — it will appear in the list automatically
|
||||
An article's directory *name* is where its publish date, tags, category, and draft status all
|
||||
live — not in separate files, and not in frontmatter. The format is:
|
||||
|
||||
`author` is derived from the directory path — no frontmatter field needed for it. Publish date,
|
||||
tags, category, and draft status are all derived from marker files, not frontmatter — this
|
||||
keeps every one of them knowable from a directory listing alone, without fetching an article's
|
||||
content, which is what makes the page-scoped loading described in "Article discovery & caching"
|
||||
below possible.
|
||||
```
|
||||
<slug>--<DDMMYYYY>--<tag1>+<tag2>+…
|
||||
```
|
||||
|
||||
- `<slug>` — the stable identifier used in the article's URL (`#post/author/slug`). Treat this
|
||||
as permanent: retagging or correcting the date later just means renaming the directory (see
|
||||
below), but changing the slug changes the article's URL.
|
||||
- `<DDMMYYYY>` — the publish date, 8 digits, no separators (e.g. `16062026` for 16 June 2026).
|
||||
- `<tag1>+<tag2>+…` — zero or more tag ids, joined with `+`. A tag id matching `expa-YYYY` is
|
||||
treated as the article's "year"/category, shown as a badge linking to `#category/expa-YYYY`;
|
||||
every other tag id is a normal topic tag, shown as a pill linking to `#tag/tag-id`. Omit this
|
||||
whole segment (and its leading `--`) if the article has no tags.
|
||||
|
||||
A leading underscore on the whole directory name — `_<slug>--...` — marks the article as a
|
||||
**draft**: it's excluded from the list, search, and pagination, but still reachable directly by
|
||||
URL. Remove the underscore to publish it.
|
||||
|
||||
Steps:
|
||||
|
||||
1. Create the author directory if it doesn't exist: `cmjs/articles/author/`
|
||||
2. Create the post directory using the naming convention above, e.g.
|
||||
`cmjs/articles/author/my-article--16062026--kometa+expa-2026/`
|
||||
3. Create the Markdown file inside, named after the slug: `my-article.md`
|
||||
4. Optionally add a cover image beside it too, e.g. `photo.jpg`, and reference it from
|
||||
frontmatter with `image: photo.jpg`. Images live next to the article that uses them, not in a
|
||||
shared folder.
|
||||
5. Add a frontmatter block at the top of the Markdown file
|
||||
6. Write your content — it will appear in the list automatically
|
||||
|
||||
To change an article's date, tags, or draft status later, rename its directory — the article's
|
||||
`slug.md` filename and content don't need to change. `author` is derived from the directory path
|
||||
— no frontmatter field needed for it either. Because every one of these is encoded in the
|
||||
directory name, discovering an article's metadata never requires fetching anything beyond the
|
||||
directory listing that already reveals it — no further per-article request at all, which is what
|
||||
makes the "Article discovery & caching" section below possible.
|
||||
|
||||
### Tag labels
|
||||
|
||||
A `.tag` marker's filename is just an id (e.g. `kometa`) — the human-readable label shown in
|
||||
the sidebar and on tag pills comes from `cmjs/blog-config/tags.json`:
|
||||
A tag in the directory name is just an id (e.g. `kometa`) — the human-readable label shown on
|
||||
tag pills comes from `cmjs/blog-config/tags.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -149,7 +160,7 @@ Your markdown content here...
|
||||
|
||||
Reading time is computed automatically from the article word count — no frontmatter field
|
||||
needed. `date`, `tags`, and `draft` frontmatter fields, if present, are ignored — superseded by
|
||||
the `.date`/`.tag`/`draft` marker files described above.
|
||||
the directory name described above.
|
||||
|
||||
## Author profiles
|
||||
|
||||
@@ -350,44 +361,37 @@ server {
|
||||
## Article discovery & caching
|
||||
|
||||
Discovery happens in two phases. **Indexing** crawls `cmjs/articles/` client-side to build a
|
||||
lightweight "stub" for every article — author directories, then each author's post directories
|
||||
along with their `mtime`, then — only for directories whose `mtime` has changed since the last
|
||||
visit — that directory's own listing, read for its `.tag`/`.date`/`draft` markers. This alone is
|
||||
enough to sort, filter, paginate, and populate the sidebar's tag/year counts, without ever
|
||||
fetching an article's actual Markdown. **Hydration** is the separate, deferred step that fetches
|
||||
a specific article's content (title, description, cover image, reading time) — it only ever runs
|
||||
for articles about to be rendered: the current page's cards (`hydrateArticles()` in
|
||||
`cmjs/js/discovery.js`), a single visited article, or (since full-text search needs real content)
|
||||
every result of a search. A large site's list is therefore fast to show regardless of how many
|
||||
articles it has — only ~20 articles' worth of content is ever fetched per page view.
|
||||
lightweight "stub" for every article — the author directories, then each author's post
|
||||
directories along with their `mtime`. That's it: because a post directory's *name* already
|
||||
encodes its slug, date, tags, and draft status (see "Adding an article" above), that single
|
||||
listing is enough to sort, filter, paginate, and populate the sidebar's tag/year counts, without
|
||||
any further request per article and without ever fetching an article's actual Markdown.
|
||||
**Hydration** is the separate, deferred step that fetches a specific article's content (title,
|
||||
description, cover image, reading time) — it only ever runs for articles about to be rendered:
|
||||
the current page's cards (`hydrateArticles()` in `cmjs/js/discovery.js`), a single visited
|
||||
article, or (since full-text search needs real content) every result of a search. A large site's
|
||||
list is therefore fast to show regardless of how many articles it has — indexing costs one
|
||||
request per author, full stop, and only ~20 articles' worth of content is ever fetched per page
|
||||
view.
|
||||
|
||||
- **Cache** — every resolved stub is cached per post directory in IndexedDB (`cmjs/js/cache.js`),
|
||||
- **Cache** — hydrated articles are cached per post directory in IndexedDB (`cmjs/js/cache.js`),
|
||||
keyed by directory path and invalidated by comparing the live `mtime` against the cached one —
|
||||
as soon as that comparison matches, indexing reuses the cached entry directly and skips the
|
||||
listing fetch too, whether or not the article has ever actually been hydrated. Hydration
|
||||
re-caches the same entry with the fetched content added, so once *any* visit renders an
|
||||
article, every later visit skips its content fetch as well, until its directory's `mtime`
|
||||
changes. Indexing reads the entire cache once up front (`getAllCacheEntries()`, kicked off in
|
||||
parallel with the author/post-directory requests) rather than a separate IndexedDB lookup per
|
||||
directory, so a warm cache doesn't cost hundreds of individual reads. If IndexedDB is
|
||||
unavailable (private browsing, disabled storage), discovery transparently falls back to
|
||||
fetching everything live on every visit.
|
||||
once *any* visit renders an article, every later visit skips its content fetch too, until its
|
||||
directory is renamed (a rename changes the cache key, so a retagged/redated article is treated
|
||||
as new — see "Adding an article" above) or its `.md` file's own mtime changes. Indexing reads
|
||||
the entire cache once up front (`getAllCacheEntries()`, kicked off in parallel with the
|
||||
author/post-directory requests) so a warm cache doesn't cost an IndexedDB read per article. If
|
||||
IndexedDB is unavailable (private browsing, disabled storage), discovery transparently falls
|
||||
back to fetching everything live on every visit.
|
||||
- **Cache schema version** — the cached records store the full derived article shape, not raw
|
||||
content, so a code change to how articles are derived (in `cmjs/js/discovery.js`) can make old
|
||||
cached entries stale in *structure*, not just content — and a directory's `mtime` won't catch
|
||||
that. Bump `CACHE_SCHEMA_VERSION` in `cmjs/js/cache.js` whenever that shape changes; it's baked
|
||||
into the IndexedDB database name, so a bump invalidates every cached entry on the next deploy.
|
||||
- **Concurrency** — `MAX_CONCURRENT_REQUESTS` in `cmjs/js/discovery.js` (default `10`) caps how
|
||||
many requests (indexing listings or hydration content fetches) run at once; raise it for a
|
||||
fast/local nginx, lower it for a slow or shared/production instance.
|
||||
- **Progressive indexing** — on a large site, the article list doesn't wait for the entire tree
|
||||
to finish indexing before showing anything: `discoverArticles()` streams incremental stub
|
||||
results via `onArticlesProgress()`, so the list (and pagination, and the sidebar tag/year
|
||||
counts) render as soon as the first articles are indexed and keep filling in while the rest
|
||||
index in the background — with a small "still indexing…" note shown until it's done. Indexing
|
||||
also visits directories with the most recent `mtime` first (a best-effort heuristic, not the
|
||||
final display order — that's always the authoritative `.date` marker), so the first results
|
||||
tend to already be close to correct.
|
||||
many hydration (content) fetches run at once — the only per-article requests left, since
|
||||
indexing no longer needs one; raise it for a fast/local nginx, lower it for a slow or shared/
|
||||
production instance.
|
||||
- **Prefetching** — once the current page has fully rendered, the next page's articles are
|
||||
hydrated in the background (fire-and-forget), so clicking "next" is usually an instant cache
|
||||
hit rather than a fresh round-trip.
|
||||
@@ -398,13 +402,17 @@ articles it has — only ~20 articles' worth of content is ever fetched per page
|
||||
list, and the page count can shift while hydration is still filling in matches, settling once
|
||||
it's done.
|
||||
- **Crawl health** — if every author's directory listing fails outright, or very few of the
|
||||
known post directories loaded, a small warning banner is shown above the article list instead
|
||||
of silently rendering an empty page (see `getCrawlStats()` / `renderCrawlHealthWarning()` in
|
||||
`cmjs/js/discovery.js` / `cmjs/js/views.js`).
|
||||
- **Design rationale** — the full reasoning (wave-based crawling, recency prioritisation,
|
||||
HTML-autoindex fallback, and long-term "deploy and forget" constraints) is documented in
|
||||
known post directories parsed successfully (a malformed directory name, or a total breakage
|
||||
like the wrong origin or CORS misconfigured), a small warning banner is shown above the article
|
||||
list instead of silently rendering an empty page (see `getCrawlStats()` /
|
||||
`renderCrawlHealthWarning()` in `cmjs/js/discovery.js` / `cmjs/js/views.js`).
|
||||
- **Design rationale** — the original crawler design (wave-based crawling with a per-article
|
||||
listing fetch, recency prioritisation, HTML-autoindex fallback, long-term "deploy and forget"
|
||||
constraints) is documented in
|
||||
[`.claude/nginx-sitemap-crawler-plan.md`](.claude/nginx-sitemap-crawler-plan.md), which ships
|
||||
alongside the code.
|
||||
alongside the code. The per-article listing fetch it describes no longer exists (metadata moved
|
||||
into the directory name instead), but the rest of that reasoning — the caching model, the
|
||||
longevity constraints, the HTML fallback — still applies.
|
||||
|
||||
## Running locally
|
||||
|
||||
|
||||
Whitespace-only changes.
Binary file not shown.
|
Before Width: | Height: | Size: 223 KiB |
@@ -1,54 +0,0 @@
|
||||
---
|
||||
title: Hello World
|
||||
description: A sample article to demonstrate the CMS.
|
||||
image: hello-world-preview.webp
|
||||
draft: false
|
||||
---
|
||||
|
||||
Welcome to **cmjs** — a minimal client-side CMS powered by Markdown.
|
||||
|
||||
## How it works
|
||||
|
||||
Articles are plain Markdown files stored in the `articles/` directory. Each file carries its own metadata via YAML frontmatter, similar to [Hugo](https://gohugo.io/):
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: My Article Title
|
||||
description: A short summary shown in the article list.
|
||||
draft: false
|
||||
---
|
||||
```
|
||||
|
||||
The app fetches the directory listing from the server, reads frontmatter from each `.md` file, and renders the list. Clicking an article fetches the file and renders the Markdown body client-side using [marked.js](https://marked.js.org/).
|
||||
|
||||
## Adding an article
|
||||
|
||||
1. Create a new `.md` file in `articles/`
|
||||
2. Add a frontmatter block at the top
|
||||
3. Write your content below the closing `---`
|
||||
|
||||
No rebuild, no CMS dashboard — the article appears in the list automatically.
|
||||
|
||||
## Frontmatter fields
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------|----------|-------------|
|
||||
| `title` | yes | Display name shown in the list and article header |
|
||||
| `date` | yes | ISO date (`YYYY-MM-DD`), used for sorting |
|
||||
| `description` | no | Short summary shown under the title in the list |
|
||||
| `draft` | no | Set to `true` to hide from the article list |
|
||||
|
||||
## Markdown features
|
||||
|
||||
Markdown renders fully — headings, lists, **bold**, _italic_, `inline code`, and fenced code blocks:
|
||||
|
||||
```javascript
|
||||
// Frontmatter is parsed with a small built-in regex parser
|
||||
function parseFrontmatter(raw) {
|
||||
const match = raw.match(/^---\r?\n([\s\S]*?)\r?\n---\r?\n([\s\S]*)$/);
|
||||
if (!match) return { meta: {}, content: raw };
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
> Articles are just files. No database, no server-side code, no login.
|
||||
+100
-128
@@ -4,20 +4,17 @@ import { mapWithConcurrency } from './concurrency.js';
|
||||
import { getAllCacheEntries, setCacheEntry, pruneCache } from './cache.js';
|
||||
import { t } from './i18n.js';
|
||||
|
||||
// Caps simultaneous per-article requests — both the cheap per-directory
|
||||
// listing fetch every article needs at index time (see "Wave 3" below) and
|
||||
// the markdown content fetch, which only ever runs for articles actually
|
||||
// being rendered (see hydrateArticle()) — so a large site doesn't burst
|
||||
// hundreds of requests at once. Raise it for a fast/local nginx, lower it
|
||||
// for a slow or shared/production instance — see
|
||||
// .claude/nginx-sitemap-crawler-plan.md §2a for the full rationale.
|
||||
// Caps simultaneous markdown content fetches during hydration (see
|
||||
// hydrateArticle()/hydrateArticles()) — the only per-article requests left,
|
||||
// since indexing no longer needs a per-article listing fetch at all (every
|
||||
// stub field is encoded in its directory's own name — see buildStub()). Raise
|
||||
// it for a fast/local nginx, lower it for a slow or shared/production
|
||||
// instance — see .claude/nginx-sitemap-crawler-plan.md §2a for the original
|
||||
// rationale (written when this also governed indexing requests).
|
||||
const MAX_CONCURRENT_REQUESTS = 10;
|
||||
|
||||
const MONTH_ABBREVIATIONS = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];
|
||||
|
||||
// A slug directory's own `<DD-MM-YYYY>.date` marker file (see README).
|
||||
const DATE_FILE_PATTERN = /^(\d{2})-(\d{2})-(\d{4})\.date$/;
|
||||
|
||||
function parseHtmlDirectoryListing(raw) {
|
||||
const entries = [];
|
||||
const seen = new Set();
|
||||
@@ -152,73 +149,76 @@ function excerpt(content) {
|
||||
return `${text.slice(0, 277).replace(/\s+\S*$/, '')}…`;
|
||||
}
|
||||
|
||||
// A single `<DD-MM-YYYY>.date` marker file is the sole authoritative source
|
||||
// of an article's publish date — no frontmatter fallback, so this is always
|
||||
// resolvable from a directory listing alone, without fetching content.
|
||||
function parseDateFromFiles(files) {
|
||||
const matches = files.map(file => file.name.match(DATE_FILE_PATTERN)).filter(Boolean);
|
||||
if (matches.length !== 1) return null;
|
||||
const [, day, month, year] = matches[0];
|
||||
return `${year}-${month}-${day}`;
|
||||
// A post directory's own name is the sole authoritative source of its
|
||||
// slug, publish date, tags/category, and draft status — see the README's
|
||||
// "Adding an article" section. Format: `<slug>--<DDMMYYYY>[--tag1+tag2+…]`,
|
||||
// optionally prefixed with `_` for draft. Splitting on the whole name is
|
||||
// deliberate: a slug is expected not to contain "--" itself, the same way
|
||||
// it's already expected not to contain "/". Returns null for any directory
|
||||
// name that doesn't match — callers treat that as a silent skip, e.g. a
|
||||
// stray directory that isn't a post at all.
|
||||
function parseSlugDirName(name) {
|
||||
const draft = name.startsWith('_');
|
||||
const core = draft ? name.slice(1) : name;
|
||||
const [slug, dateSegment, tagsSegment] = core.split('--');
|
||||
if (!slug || !dateSegment || !/^\d{8}$/.test(dateSegment)) return null;
|
||||
const day = dateSegment.slice(0, 2);
|
||||
const month = dateSegment.slice(2, 4);
|
||||
const year = dateSegment.slice(4, 8);
|
||||
return {
|
||||
slug,
|
||||
date: `${year}-${month}-${day}`,
|
||||
tags: tagsSegment ? tagsSegment.split('+').filter(Boolean) : [],
|
||||
draft,
|
||||
};
|
||||
}
|
||||
|
||||
// Everything an article card/filter/sort needs that's derivable from its
|
||||
// slug directory's own *listing* alone — no markdown fetch. This is what
|
||||
// runs for every article at index time; the (expensive) content fetch is
|
||||
// deferred to hydrateArticle(), which only ever runs for articles actually
|
||||
// being rendered. Returns null on any failure (missing markdown file,
|
||||
// network error) — the caller treats that as a silent skip, matching the
|
||||
// rest of this module's approach.
|
||||
async function buildStub(dir) {
|
||||
try {
|
||||
const entries = await fetchJson(dir.root);
|
||||
const files = entries.filter(entry => entry.type === 'file');
|
||||
const tagIds = files.filter(entry => entry.name.endsWith('.tag')).map(entry => entry.name.replace(/\.tag$/, ''));
|
||||
const mdFiles = files.filter(entry => entry.name.endsWith('.md') && !entry.name.startsWith('_'));
|
||||
const mdFile = mdFiles.find(entry => entry.name === `${dir.slug}.md`)
|
||||
|| mdFiles.find(entry => entry.name === 'index.md')
|
||||
|| (mdFiles.length === 1 ? mdFiles[0] : null);
|
||||
if (!mdFile) return null;
|
||||
// "Category" isn't a separate concept anymore — it's just whichever
|
||||
// tag id (if any) looks like an expedition year.
|
||||
const category = tagIds.find(id => /^expa-20\d{2}$/i.test(id)) || null;
|
||||
// Everything an article card/filter/sort needs, parsed entirely from its
|
||||
// directory's own name and its listing entry's mtime — no fetch of any
|
||||
// kind. This is what runs for every article at index time; the (expensive)
|
||||
// content fetch is deferred to hydrateArticle(), which only ever runs for
|
||||
// articles actually being rendered. Returns null if the directory name
|
||||
// doesn't match the naming convention (see parseSlugDirName) — the caller
|
||||
// treats that as a silent skip, matching the rest of this module's
|
||||
// approach.
|
||||
function buildStub(author, dirName, mtime, authorRoot) {
|
||||
const parsed = parseSlugDirName(dirName);
|
||||
if (!parsed) return null;
|
||||
const { slug, date, tags, draft } = parsed;
|
||||
const root = `${authorRoot}${encodeURIComponent(dirName)}/`;
|
||||
// "Category" isn't a separate concept anymore — it's just whichever tag
|
||||
// id (if any) looks like an expedition year.
|
||||
const category = tags.find(id => /^expa-20\d{2}$/i.test(id)) || null;
|
||||
|
||||
return {
|
||||
author: dir.author,
|
||||
slug: dir.slug,
|
||||
hash: `${dir.author}/${dir.slug}`,
|
||||
root: dir.root,
|
||||
mtime: dir.mtime,
|
||||
path: `${dir.root}${encodeURIComponent(mdFile.name)}`,
|
||||
category,
|
||||
expa: category ? Number(category.replace(/^expa-/i, '')) : null,
|
||||
date: parseDateFromFiles(files),
|
||||
tags: tagIds,
|
||||
// A `draft` marker file (no extension, exact name) is the sole
|
||||
// source of publish status — like `.tag`/`.date`, this needs to
|
||||
// be knowable from the listing alone, not from a content fetch.
|
||||
published: !files.some(entry => entry.name === 'draft'),
|
||||
hydrated: false,
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
author,
|
||||
slug,
|
||||
hash: `${author}/${slug}`,
|
||||
root,
|
||||
mtime,
|
||||
path: `${root}${encodeURIComponent(slug)}.md`,
|
||||
category,
|
||||
expa: category ? Number(category.replace(/^expa-/i, '')) : null,
|
||||
date,
|
||||
tags,
|
||||
published: !draft,
|
||||
hydrated: false,
|
||||
};
|
||||
}
|
||||
|
||||
// Cache-aware: a directory whose mtime matches its cached entry (stub *or*
|
||||
// already-hydrated, either is fine here) is reused directly, skipping the
|
||||
// listing fetch entirely; otherwise builds a fresh stub from the listing
|
||||
// and caches that too, so a repeat visit skips the listing fetch even for
|
||||
// an article nothing has hydrated yet — not just already-viewed ones.
|
||||
// `cacheSnapshot` is the whole cache read once up front (see runCrawl) —
|
||||
// an in-memory lookup here instead of a separate IndexedDB read per
|
||||
// directory, which matters once there are hundreds of them.
|
||||
async function resolveStub(dir, cacheSnapshot) {
|
||||
const cached = cacheSnapshot.get(dir.root);
|
||||
if (cached && dir.mtime != null && cached.mtime === dir.mtime) return cached.article;
|
||||
const stub = await buildStub(dir);
|
||||
if (stub) await setCacheEntry(dir.root, { mtime: dir.mtime, article: stub });
|
||||
return stub;
|
||||
// Cache-aware: a directory whose mtime matches its cached entry's mtime
|
||||
// reuses that entry directly — worthwhile only when it's already hydrated
|
||||
// (skipping a real content re-fetch); a merely-parsed stub is free to
|
||||
// rebuild, so there's no point caching one just to skip re-parsing a
|
||||
// string. `cacheSnapshot` is the whole cache read once up front (see
|
||||
// runCrawl) — an in-memory lookup here instead of a separate IndexedDB read
|
||||
// per directory, which mattered when this also had to guard a listing
|
||||
// fetch; kept anyway since it's still the same one cheap lookup.
|
||||
function resolveStub(author, dirName, mtime, authorRoot, cacheSnapshot) {
|
||||
const root = `${authorRoot}${encodeURIComponent(dirName)}/`;
|
||||
const cached = cacheSnapshot.get(root);
|
||||
if (cached && mtime != null && cached.mtime === mtime && cached.article?.hydrated) return cached.article;
|
||||
return buildStub(author, dirName, mtime, authorRoot);
|
||||
}
|
||||
|
||||
// Fields only derivable from an article's actual markdown content (not its
|
||||
@@ -274,10 +274,7 @@ export async function hydrateArticle(stub) {
|
||||
// just claims the next unclaimed index) — since discoverArticles() already
|
||||
// returns stubs sorted newest-first by the authoritative `date` field, that
|
||||
// means hydration (and thus which search results fill in first) is already
|
||||
// newest-first too, for free. Unlike the indexing crawl's mtime-based
|
||||
// priority order (see mtimeSortValue), no heuristic is needed here: by the
|
||||
// time anything calls this, the real publish date is already known for
|
||||
// every stub, so there's nothing left to guess at.
|
||||
// newest-first too, for free.
|
||||
export function hydrateArticles(stubs, onProgress) {
|
||||
let resolved = 0;
|
||||
return mapWithConcurrency(stubs, MAX_CONCURRENT_REQUESTS, async stub => {
|
||||
@@ -288,23 +285,13 @@ export function hydrateArticles(stubs, onProgress) {
|
||||
});
|
||||
}
|
||||
|
||||
// Best-effort recency signal used only to decide crawl *order* (see below) —
|
||||
// never for display order, which always uses the authoritative `date`
|
||||
// marker once the crawl finishes. `Date.parse` handles both the RFC-1123
|
||||
// string nginx's JSON autoindex emits and the ISO string the HTML-fallback
|
||||
// parser produces; an unparseable/missing mtime sorts last.
|
||||
function mtimeSortValue(dir) {
|
||||
const parsed = dir.mtime ? Date.parse(dir.mtime) : NaN;
|
||||
return Number.isNaN(parsed) ? -Infinity : parsed;
|
||||
}
|
||||
|
||||
async function runCrawl() {
|
||||
partialResolved = [];
|
||||
|
||||
// Independent of waves 1/2 below — kicked off in parallel with them and
|
||||
// only awaited once wave 3 actually needs it, so the cache read's
|
||||
// latency is hidden behind the (already-required) directory fetches
|
||||
// rather than adding to the total.
|
||||
// only awaited once it's actually needed, so its latency is hidden
|
||||
// behind the (already-required) directory fetches rather than adding
|
||||
// to the total.
|
||||
const cacheSnapshotPromise = getAllCacheEntries();
|
||||
|
||||
// Wave 1 (root) — always fetched live: the author directories.
|
||||
@@ -312,64 +299,49 @@ async function runCrawl() {
|
||||
const authors = authorEntries.filter(entry => entry.type === 'directory' && !entry.name.startsWith('.'));
|
||||
|
||||
// Wave 2 — always fetched live, in full, for every author (not just the
|
||||
// current page): each author's slug directories, with their mtimes. This
|
||||
// is what builds the flat, cross-author list that drives global sorting
|
||||
// and pagination — see the plan doc's "keep the flat feed" decision.
|
||||
// One author's listing failing (network blip, deleted mid-crawl) is
|
||||
// silently skipped rather than failing every other author's list too —
|
||||
// consistent with this module's usual per-item failure handling (see
|
||||
// buildStub) — but is still counted, so a *total* breakage (wrong
|
||||
// origin, CORS) remains visible via getCrawlStats() instead of just
|
||||
// silently producing an empty article list.
|
||||
// current page): each author's post directories, with their mtimes.
|
||||
// This is what builds the flat, cross-author list that drives global
|
||||
// sorting and pagination — see the plan doc's "keep the flat feed"
|
||||
// decision. It's also now the *only* per-directory network request
|
||||
// indexing ever needs: every stub field (slug, date, tags, category,
|
||||
// draft status) is parsed straight out of each directory's own name
|
||||
// (see buildStub/parseSlugDirName) — there's no more separate
|
||||
// per-article listing fetch the way there used to be. One author's
|
||||
// listing failing (network blip, deleted mid-crawl) is silently
|
||||
// skipped rather than failing every other author's list too, but is
|
||||
// still counted, so a *total* breakage (wrong origin, CORS) remains
|
||||
// visible via getCrawlStats() instead of just silently producing an
|
||||
// empty article list.
|
||||
let authorsFailed = 0;
|
||||
const slugDirLists = await Promise.all(authors.map(async author => {
|
||||
const authorDirLists = await Promise.all(authors.map(async author => {
|
||||
const authorRoot = `${BASE_URL}articles/${encodeURIComponent(author.name)}/`;
|
||||
try {
|
||||
const entries = await fetchJson(authorRoot);
|
||||
return entries
|
||||
.filter(entry => entry.type === 'directory' && entry.name !== '_history')
|
||||
.map(entry => ({
|
||||
author: author.name,
|
||||
slug: entry.name,
|
||||
mtime: entry.mtime ?? null,
|
||||
root: `${authorRoot}${encodeURIComponent(entry.name)}/`,
|
||||
}));
|
||||
.map(entry => ({ author: author.name, dirName: entry.name, mtime: entry.mtime ?? null, authorRoot }));
|
||||
} catch {
|
||||
authorsFailed += 1;
|
||||
return [];
|
||||
}
|
||||
}));
|
||||
const slugDirs = slugDirLists.flat();
|
||||
const allDirs = authorDirLists.flat();
|
||||
|
||||
// Crawl the likeliest-recent directories first so the first page of
|
||||
// results tends to already be close to correct while the rest of a
|
||||
// large tree is still indexing in the background (see
|
||||
// onArticlesProgress). Purely a priority order, not the display order.
|
||||
const crawlOrder = [...slugDirs].sort((a, b) => mtimeSortValue(b) - mtimeSortValue(a));
|
||||
// Every stub is parsed synchronously from data already in hand — no
|
||||
// per-directory fetch, so no need for mapWithConcurrency here (that's
|
||||
// still used by hydrateArticles(), which does real network fetches).
|
||||
const cacheSnapshot = await cacheSnapshotPromise;
|
||||
|
||||
// Wave 3 — per slug directory, cache-aware and concurrency-limited: a
|
||||
// directory whose mtime matches its cached entry (stub or hydrated)
|
||||
// skips the listing fetch entirely; otherwise just its listing (for
|
||||
// .tag/.date/draft markers) — not its markdown content, which is
|
||||
// deferred to hydrateArticle() until something actually renders it.
|
||||
let loaded = 0;
|
||||
let resolvedCount = 0;
|
||||
await mapWithConcurrency(crawlOrder, MAX_CONCURRENT_REQUESTS, async dir => {
|
||||
const stub = await resolveStub(dir, cacheSnapshot);
|
||||
for (const dir of allDirs) {
|
||||
const stub = resolveStub(dir.author, dir.dirName, dir.mtime, dir.authorRoot, cacheSnapshot);
|
||||
if (stub) {
|
||||
loaded += 1;
|
||||
partialResolved.push(stub);
|
||||
}
|
||||
// Throttled rather than on every single item — with a large tree
|
||||
// that would mean hundreds of list re-renders for little benefit.
|
||||
resolvedCount += 1;
|
||||
if (resolvedCount === 1 || resolvedCount % 5 === 0 || resolvedCount === crawlOrder.length) notifyProgress(false);
|
||||
return stub;
|
||||
});
|
||||
}
|
||||
|
||||
await pruneCache(slugDirs.map(dir => dir.root));
|
||||
crawlStats = { total: slugDirs.length, loaded, authorsTotal: authors.length, authorsFailed };
|
||||
await pruneCache(allDirs.map(dir => `${dir.authorRoot}${encodeURIComponent(dir.dirName)}/`));
|
||||
crawlStats = { total: allDirs.length, loaded, authorsTotal: authors.length, authorsFailed };
|
||||
|
||||
const finalArticles = sortAndFilterPublished(partialResolved);
|
||||
notifyProgress(true);
|
||||
|
||||
+4
-4
@@ -175,8 +175,8 @@ function renderCard(article) {
|
||||
const author = field('author');
|
||||
author.textContent = article.authorName || article.author;
|
||||
author.href = `#author/${encodeURIComponent(article.author)}`;
|
||||
// A slug directory only counts as a "category" when one of its .tag
|
||||
// markers happens to name an expedition year — not every article has one.
|
||||
// A post only counts as a "category" when one of its directory-name
|
||||
// tags happens to name an expedition year — not every article has one.
|
||||
setField(field('category'), article.category, (category, value) => {
|
||||
category.textContent = t('yearLabel', { year: article.expa });
|
||||
category.href = `#category/${encodeURIComponent(value)}`;
|
||||
@@ -458,8 +458,8 @@ export async function loadArticle(hash) {
|
||||
const element = document.importNode(templates.article, true);
|
||||
const field = name => element.querySelector(`[data-field="${name}"]`);
|
||||
field('title').textContent = meta.title || article.slug;
|
||||
// Publish date comes from the slug directory's `.date` marker file
|
||||
// (already resolved onto `article.date`), not the raw frontmatter.
|
||||
// Publish date comes from the post directory's own name (already
|
||||
// resolved onto `article.date`), not the raw frontmatter.
|
||||
setField(field('date'), article.date, (time, value) => { time.dateTime = String(value); time.textContent = formatDate(value); });
|
||||
setField(field('description'), meta.description, (description, value) => { description.textContent = value; });
|
||||
setField(field('reading-time'), readingTime, (time, value) => { time.textContent = t('readingTime', { minutes: value }); });
|
||||
|
||||
Reference in new issue
Block a user