Melody Genealogy — Crate-Digging the Public Domain with Local Search

You are an ethnomusicologist and a music-IR engineer. Build me a fully local melody-genealogy engine, not a vibes graph.
1. Pull 200 to 500 public-domain recordings from the Library of Congress and Internet Archive 78s. Verify rights recording-by-recording: composition and recording are separate. Drop anything not clearly public domain.
2. Extract transposition-invariant pitch contours (CREPE or pYIN, quantized to intervals) and audio embeddings (a CLAP encoder). Store both in ChromaDB in the SentrySearch offline layout.
3. Cluster both representations. Score them against a labelled ground-truth set of known families from the Child and Roud indices, including 'Shady Grove' / 'Matty Groves' (Child #81) and the 'Hog-Eye Man' tunes. Report precision, recall and adjusted Rand index.
4. Only if families cluster above chance, build a lineage graph in NetworkX with timestamped links to each source recording.
5. Package one cluster as a public-domain sample pack with a per-sample provenance sheet clearing composition and recording.
6. Pre-register the falsifier: if known families do not cluster above chance, the hand method stays the only method.
Return the corpus manifest, embeddings, cluster scores, the lineage graph, and the cleared sample pack.

ROLE
You are an ethnomusicologist and a music-information-retrieval engineer. You test similarity against ground truth before you draw a family tree, and you clear rights before you call anything a sample.

CONTEXT
- @chancemccoymusic traces Doc Watson's 'Shady Grove' in D-Dorian clawhammer to Jean Ritchie's father in Viper, Kentucky, and back to Child Ballad #81, 'Matty Groves'; and 'Sally in the Garden' to Crockett's Kentucky Mountaineers' 1929 Brunswick 291, related to the Ozark 'Hog-Eye Man' tunes. He does this by ear, one tune at a time.
- @seanofthesouth: 'Shady Grove' crossed the Atlantic as a Renaissance-era song.
- @dimestoreradio: Lomax field footage of Nimrod Workman at home.
- @6ab3: Library of Congress, Internet Archive 78s and the Prelinger films as public-domain sources.
- @ssrajadh/sentrysearch: a 100% local video-search engine, type a query and it returns timestamps, using Qwen3-VL embeddings and ChromaDB. The layout to copy for audio.
- @lizthedeveloper: the Platonic Representation Hypothesis, shared embedding spaces across models, framed as 'AI telepathy'. A hypothesis to test, not assume.
- @baauer: Passion Pit's 'Sleepyhead' sampled Mary O'Hara's 1958 'Oro Mo Bhaidin'. @aklmadz: where producers hunt samples.
- @lilithsomething and @kvashenaya: 'The Winner Takes It All' in Mandarin and Ievan Polkka in Russian, melody kept, lyrics re-written, lineage edges.

HARD CONSTRAINTS
- Rights, recording by recording. A public-domain composition does not make a given recording public domain. Verify composition and recording separately, and drop anything not clearly cleared.
- Fully local, like SentrySearch. No recording leaves the machine.
- Transposition-invariant melody: compare interval sequences, not raw pitch.
- Prove families cluster above chance before building any lineage graph.
- Every lineage edge links to a timestamped source recording. No uncited edges.

DELIVERABLES
1. corpus/: a fetcher and manifest.csv (source, url, year, composition rights, recording rights, cleared y/n) for 200 to 500 public-domain recordings.
2. index/: pitch contours (CREPE or pYIN, quantized to intervals) and CLAP audio embeddings, both in ChromaDB in the SentrySearch offline layout.
3. eval/: a labelled ground-truth set of known families from the Child and Roud indices, and a score of the clustering by precision, recall and adjusted Rand index.
4. lineage/: a NetworkX graph and a force-directed web view, each edge carrying a timestamped link to its source recording, built only if the eval passes.
5. pack/: one cluster as a sample pack with a per-sample provenance sheet clearing composition and recording.
6. report.md.

BUILT-IN GAP ANALYSIS
List recordings dropped for unclear rights, families with no ground-truth label, edges with no timestamped source, and clusters that do not beat chance. Fill what you can. Mark the rest UNFILLED.

SELF-QUESTIONS
Is each recording's rights cleared, not just the composition's? Do 'Shady Grove' and 'Matty Groves' actually cluster together? Is my melody representation transposition-invariant? Does every lineage edge cite a source?

STOP
Stop when the clustering is scored against ground truth and, if it passes, the lineage graph and one cleared pack are built. Stop and report the falsifier if families do not beat chance.

RETURN
Return corpus/, index/, eval/, lineage/, pack/ and report.md with the sections Corpus, Index, Evaluation, Lineage, Sample Pack and Gaps.

Pull 200 to 500 recordings from public-domain sources: Library of Congress collections (including the Lomax field recordings and the Nimrod Workman lineage @dimestoreradio shows), Internet Archive 78-rpm collections, and Prelinger film audio. Include the anchors @chancemccoymusic names: 'Shady Grove' versions (Doc Watson, Jean Ritchie), 'Matty Groves', 'Sally in the Garden' (Crockett's Kentucky Mountaineers, Brunswick 291), and 'Hog-Eye Man' tunes.
For each recording, verify rights twice: the composition (often public domain if pre-1929 in the US) and the specific recording (a 1929 recording's rights can differ). Record both in manifest.csv with the rights source, and set cleared=y only when both are clear.
Fetch locally and keep audio on the machine. Record the duration and recording year of each item, since the lineage graph orients edges by date. Keep a cleared-count so you know how many recordings are actually usable in the sample pack, separate from the full corpus.

For each recording, extract the monophonic melody with CREPE or pYIN (via librosa). Convert the pitch track to a sequence of intervals (semitone steps between successive notes) so the representation is transposition-invariant, which matters because the same tune appears in different keys and modal tunings, like the D-Dorian 'Shady Grove'.
Separately, compute an audio embedding with an open CLAP audio encoder for timbral and overall similarity.
Store both representations in ChromaDB, following the SentrySearch (ssrajadh/sentrysearch) fully-local layout: one collection for melody-interval vectors, one for audio embeddings, with metadata linking back to the recording and timestamp.
Keep everything offline.
Handle polyphonic or noisy 78s by flagging low-confidence melody extractions rather than trusting them. Store the raw pitch track alongside the quantized intervals for debugging, but index only the intervals. Note the sample rate and any resampling, since 78-rpm transfers vary and pitch extraction is sensitive to speed errors.

Build a labelled ground-truth set: assign known family labels from the Child ballad numbers and the Roud Folk Song Index to the recordings where the literature establishes them. At minimum: 'Shady Grove' and 'Matty Groves' as Child #81, the 'Hog-Eye Man' group, and a few unrelated control tunes that should not cluster together.
Cluster both representations (interval contours and audio embeddings) with HDBSCAN or agglomerative clustering.
Score the clustering against the ground truth: precision, recall, and adjusted Rand index. Specifically check: do 'Shady Grove' and 'Matty Groves' land in the same cluster? Do the controls stay apart?
State the falsifier up front: if known families do not cluster above chance, the embedding search is noise and the hand method stays the only method.

Proceed only if step 3 showed known families clustering above chance. If it did not, skip to the report and record the falsifier result.
Build a lineage graph in NetworkX. Nodes are recordings or tune variants. Edges connect variants that are melodically close (by interval-contour distance), with the edge weight being the similarity.
Every edge must carry a timestamped link back to the source recording that supports it, in the SentrySearch spirit of returning exact timestamps.
Orient edges by date where the recording year is known, so the graph reads as travel over time: Renaissance ballad to Appalachian standard, 1929 Brunswick to later field recordings.
Add the cross-language edges as test cases: the Mandarin 'Winner Takes It All' and Russian Ievan Polkka should connect to their source melodies despite different lyrics.
Render a force-directed web view.

Pick one cluster that clustered cleanly and whose recordings are all cleared=y in both composition and recording rights.
Cut usable sample segments (loops, one-shots, vocal phrases) from those recordings.
Write a provenance sheet per sample: the source recording, its year, the composition and recording rights basis, and the lineage position (which tune family, which known relatives). This is the opposite of crate-digging into a clearance fight, as @baauer's 'Sleepyhead' sample story warns.
Assemble the pack with the samples, the provenance sheet, and a short written family history.
Prepare a free release and define how you will measure producer pickup (downloads, uses, tags). Keep the exact in and out points for each sample so cuts are reproducible. Note the tempo and key of each sample, because producers filter on those and it makes the pack more useful.

The dive leans on @lizthedeveloper's framing of the Platonic Representation Hypothesis: different models' embedding spaces converge, so melodic similarity might be findable across sources. Test it rather than assume it.
Take the interval-contour similarity and the CLAP audio similarity for the same pairs of recordings. Correlate them. If two independent representations agree on which tunes are related, that is evidence the similarity is real structure, not an artifact of one model.
Report the correlation and where the two disagree, since disagreements are the interesting cases: same melody, different timbre (a cross-language cover), or similar timbre, different melody (a false friend).
Do not overclaim: a positive correlation supports the method, it does not prove any specific lineage edge.

Run the oracle pass:
(a) Known families cluster above chance: the precision, recall and adjusted Rand index against the Child/Roud ground truth, and the explicit 'Shady Grove' / 'Matty Groves' check.
(b) Transposition invariance of the melody representation, verified by the transpose test.
(c) Every lineage edge cites a timestamped source recording.
(d) Every sample in the pack is cleared at both the composition and recording level.
(e) The falsifier: if known families do not beat chance, say plainly the embedding search is noise and the hand method stays the only method.
Write report.md with the sections Corpus, Index, Evaluation, Lineage, Sample Pack, Platonic-Representation Check and Gaps. State whether the engine actually finds tune families. Rank the gaps by impact.

You are an independent reviewer: an ethnomusicologist plus an ML engineer. Open report.md, corpus/manifest.csv, index/, eval/, lineage/ and pack/.
1. Enumerate every claim: corpus rights, clustering scores, specific lineage edges, and sample clearances. Mark each verified, unverified or false.
2. Sample 20 corpus rows and confirm composition and recording rights are each verified, not conflated.
3. Recompute the clustering scores against the ground truth, including the Shady Grove / Matty Groves check and the controls. Is it above a random-label baseline?
4. Run the transpose test on a recording and confirm the interval contour is stable.
5. Pick 10 lineage edges and confirm each cites a real timestamped source. Confirm cross-language covers attach to the correct melody.
6. Confirm every sample in the pack is cleared at both the composition and recording level.
7. List what is missing: unclear-rights recordings, families with no label, uncited edges, clusters that do not beat chance.
8. Rank the gaps by impact. Fix the top three now. Report what you could not fix and why.
Return audit.md with the columns claim, status, evidence and fix, plus 'Remaining gaps' and an A-F grade on whether the engine finds real tune families.