Data model

This page is for anyone who wants to work with TimeCues’ files directly — to back them up, script against them, or move a corpus between installs. (If you just want to use the app, you never need to touch any of this.) The short story: everything you create is plain JSON and audio files on disk, grouped into a couple of predictable folders, and each song is keyed by its slug (its lowercased filename). The details below map out exactly where each kind of file lives and which ones are precious versus rebuildable.

The app reads from two parallel trees rooted at the repo:

data/          — user/local data. Bind-mounted in docker, gitignored.
                 All writes (annotations, song-info, BPM caches, layers) go here.
data-default/  — read-only seed dataset shipped inside the docker image.
                 CC0 audio + metadata so a fresh install has something to play.

Strict separation: demo / public-tier requests resolve only under data-default/; team / researcher / admin requests resolve only under data/. The two corpora are never merged and there is no cross-corpus fallback in either direction (see corpusForReq() in web-app/vite.config.ts).

Per-annotator scoping is via the X-Annotator-Id HTTP header sent by the browser (currently the user’s email, e.g. you@example.com).

Folders under `data/` (and the mirrored `data-default/`)

Path	Per-annotator?	Purpose
`songs/<slug>/<file>.mp3`	no	One audio file per slug folder.
`song-info/<slug>.json`	no — shared, admin-only	Per-song metadata (BPM, time signature, displayed title, license). One file per slug, used by all annotators.
`annotators/<id>.json`	n/a	One file per annotator. Saved sign-up profile (displayName, email, role, affiliation, authMethod, invitedBy). Used by the Email login pane to detect returning users and by the Team page’s invite flow.
`dataset-config.json`	n/a	Single corpus-wide config file: `peopleByEmail` access tiers (admin/researcher/team/public), `sharedCorpus`, `corpusName`, BPM/grid lock state. Legacy `adminEmails` / `teamEmails` arrays are kept for back-compat.
`annotations/auto-guess/<annotator>/<slug>.json`	yes	AutoGuess output (algorithm clusters with per-point ✓/✗/@ review).
`annotations/layers/<annotator>/<slug>.json`	yes	Every annotation the curator authored for a song — boundaries, Cues, Spans, Loops, Riff Patterns, Lyrics — in one document: ordering, rename, visibility, items. Source of truth for a solo song.
`annotations/shared/<slug>/layers.json`	no — one per team	The single annotation for a collaborative song. The folder’s existence is what makes the song collaborative; nothing else records it, so nothing can drift out of sync.
`annotations/shared/<slug>/.git/`	no	A real git repo, one commit per hand-off of the edit lease. `git log` reads as who held the song, for how long, and what changed; `git show <sha>:layers.json` is any earlier version. Created with the shared folder. Never rewritten — restoring an old version writes a new commit.
`annotations/shared-archive/<slug>-<ts>/`	no	Where a shared folder goes when a song stops being collaborative. Un-sharing is not a reason to destroy the history.
`annotations/locks/<slug>.json`	no	The edit lease for a collaborative song: holder, session, when it was taken, last heartbeat, last write. Deliberately OUTSIDE the git repo — locking and unlocking happens constantly and changes no annotation. Disposable: deleting one frees the song.
`annotations/custom/<script_name>/<annotator>/<slug>.json`	yes	Annotations produced by custom user scripts.
`algorithm-outputs/algo-clusters/<slug>.json`	no	MSAF centroid-linkage clusters per song. Regenerable from `tools/cache-algo-clusters.mjs`.
`algorithm-outputs/bpm-detections/<slug>.json`	no	BPM detector outputs (5 detectors). Regenerated by `tools/python/bpm_server.py`.
`algorithm-outputs/mir-features/<slug>.json`	no	MIR feature cache. Regenerated by `tools/python/mir_server.py`.
`algorithm-outputs/msaf/<slug>/`	no	MSAF raw outputs (`msaf-{sf,foote,cnmf,olda,bpm}.json` + `estimations.jams`). Regenerable from `tools/run_msaf_songs.py`. Created on demand.
`algorithm-outputs/msaf-batch-jams/`	no	Flat MSAF JAMS dump from a batch run (one `.jams` per song, named with original audio filename). Archive only.
`algorithm-outputs/custom/<script_name>/<slug>.json`	no	Algorithm-mode results from custom user scripts.

data-default/ ships only the parallel songs/, song-info/, and annotations/{manual,auto-guess}/ (the annotation subdirs are empty — just .gitkeep placeholders, since the seed corpus has no human labels). It also ships pre-computed Demucs stems at data-default/stems/<slug>/ so demo visitors can hear individual stems on the CC0 tracks.

Note on annotation timing: there is no separate annotations/timing/ folder. Per-song time spent is embedded as time_spent_seconds directly in each annotation JSON (manual / auto-guess), co-located with the annotation it describes.

API surface

The Vite dev server exposes each folder under a stable URL prefix that does not change with the on-disk path:

URL	On-disk
`GET/POST/DELETE /api/annotation-layers/:slug`	`<root>/annotations/layers/<annotator>/<slug>.json`, or `<root>/annotations/shared/<slug>/layers.json` when the song is collaborative
`GET/POST/DELETE /api/auto-guess-annotations/:slug`	`<root>/annotations/auto-guess/<annotator>/<slug>.json`
`GET /api/annotation-locks`, `GET/POST /api/annotation-locks/:slug`	`data/annotations/locks/<slug>.json` (`POST` actions: `lock`, `heartbeat`, `unlock`, `takeover`)
`GET/POST /api/annotation-history/:slug`	`git log` / `git show` inside `data/annotations/shared/<slug>/`
`GET /api/shared-annotations`, `POST/DELETE /api/shared-annotations/:slug`	creates or archives `data/annotations/shared/<slug>/` (admin only)
`GET/POST /api/annotation-times/:slug`	embedded `time_spent_seconds` in the above
`GET/POST/DELETE /api/song-info/:slug`	`<root>/song-info/<slug>.json`
`GET/POST /api/dataset-config`	`data/dataset-config.json`
`GET/POST /api/algo-clusters/:slug`	`data/algorithm-outputs/algo-clusters/<slug>.json`
`GET /audio/<filename>`	matches in `<root>/songs/*/`

<root> is data/ or data-default/ depending on the requester’s access tier (see strict-separation note above).

Algorithm servers (BPM, MIR, mir_eval, ruptures, Demucs stems) run as separate Python processes and are reverse-proxied by Vite — see tools/python/*.py. Analysis caches land in data/algorithm-outputs/analysis/<slug>/; Demucs stem outputs land in web-app/public/stems/<slug>/ (runtime cache, regenerated on demand).

Path constants — one source of truth per language

If you move a data folder on disk, update only these files:

TypeScript / Node: web-app/dataPaths.ts exports DATA_DIRS, DATA_FILES, DEFAULT_DATA_DIRS, API_PATHS. Imported by vite.config.ts.
Python: tools/python/paths.py exports DATA_DIR, DEFAULT_DATA_DIR, MANUAL_ANNOTATIONS_DIR, BPM_DETECTIONS_DIR, etc. Imported by all Python tools.
Plain JS (.mjs): tools/cache-algo-clusters.mjs inlines its paths with a comment pointing to dataPaths.ts as the canonical source.

What’s in git, what’s not

data/ is fully gitignored (data/* in .gitignore). Nothing under it ships with the repo. Your local annotations, song-info, annotator profiles, and dataset-config.json live on your disk only. A fresh clone has an empty (or nonexistent) data/, and the app boots into the data-default/ seed corpus with no admin and no annotations.
data-default/ is fully tracked. Audio (CC0 only), song-info/, pre-computed stems/, and empty-but-tracked annotation subdirs (via .gitkeep). Anything you add here ships in the docker image.

Source of truth vs regenerable

Source of truth (back up — not regenerable): everything under data/annotations/, data/song-info/, data/annotators/, and data/dataset-config.json. These are human-produced. This includes data/annotations/shared/ — the per-song git repos ARE the version history, and a backup that skips .git keeps the current annotation while losing every earlier one. The one disposable thing under annotations/ is locks/: leases are live state, and a restored one just means a song looks locked until someone takes it over.
Regenerable caches (re-run the producer if missing): everything under data/algorithm-outputs/ (analysis, algo-clusters, bpm-detections, mir-features, msaf, msaf-batch-jams, custom). Also web-app/public/stems/ (Demucs).