Daily data vintages from 27 Aug to 4 Nov 2026 exist only in orphaned bot commits: archive or recover them before GitHub drops them #222

Activity

1. vahid-ahmadi commented on Nov 6, 2026

   

   vahid-ahmadi

   on Nov 6, 2026

   ContributorAuthor

   More actions

   Archived: option 2 is done

   Branch data/archive-orphaned-snapshots-2026-11 → commit 77f3176, created through the git data API (no clone, no force-push, main untouched at 153cf41).

   • It is a single octopus commit with 51 parents: main 153cf41 plus all 50 bot commits below. That makes every snapshot reachable, so GitHub will not garbage-collect it.
   • Its tree is main 153cf41's tree, so checking out the branch gives you main. The data is held only through the parent pointers. It is not meant for merging, so I have not opened a PR. No workflow triggers on it (every push trigger is branches: [main]).
   • Each SHA was resolved and checked as fetchable today (gh api repos/PolicyEngine/macro/commits/<sha>). compare/<orphan>...data/archive-orphaned-snapshots-2026-11 reports ahead for the first (257548b) and the last (18442ea). 18442ea is still Refresh official data vintages #195's head. It is included so that the next force-push doesn't orphan it as well.
   • The bot rebuilt the branch on main every day, so each commit holds one day's snapshot set measured against main, not a running chain. Only the series whose data differed from main's 26 Aug latest were added. There is no snapshot for 11 Sep: that run failed with an ONS connection reset.

   The 50 commits: date, full SHA, contents

   run date	commit	parent (main)	files changed	vintage files added	series added
   2026-08-27	257548b1cbe70570f93bb35ac58b69f9d0df8610	2521580	27	6	boe 4, fred 2
   2026-08-28	62b303a47ba9d65b10b749e6f284d7c7db0baa3d	2521580	27	6	boe 4, fred 2
   2026-08-31	12e03b3084b9d5e0ac8460ee2eb360ba4c3d8252	2521580	27	6	boe 4, fred 2
   2026-09-01	0083e386b53c49ebf8bc1719627bd9a81b1dd2cc	2521580	27	6	boe 4, fred 2
   2026-09-02	87df3f5e3a8ac38e3a4df8c409afe01ff53abc6d	2521580	30	7	boe 4, fred 3
   2026-09-03	d64745994c583b378b5d0cd820930382347077b3	2521580	30	7	boe 4, fred 3
   2026-09-04	d4aabf8fed875505d32621d4ba6025834a066ea3	2521580	30	7	boe 4, fred 3
   2026-09-07	c8d59330117813bcc2ab586f403be8e01dab3ceb	2521580	37	9	boe 4, fred 5
   2026-09-08	bdf57a9a0590f760e0e5fb88bdf311f8de75fe19	2521580	37	9	boe 4, fred 5
   2026-09-09	afb7abbb4bd76ee24c97e2c2d546edd4be432b1a	2521580	37	9	boe 4, fred 5
   2026-09-10	928f135bd66d418ffc4bb238e92c5abba65dac93	2521580	37	9	boe 4, fred 5
   2026-09-14	1ffb7259570cc10d3dbb7d8ddb398721ac06b641	2521580	46	11	boe 4, fred 6, ons 1
   2026-09-15	cee2d5dc4f7b4974d704f144c618f573e469fead	2521580	56	14	boe 4, fred 6, ons 4
   2026-09-16	eb65448b0654621d8fd5fce7e3748c7551831467	2521580	63	16	boe 4, fred 6, ons 6
   2026-09-17	16b73a057e3186f726cae346e3f808b63d707ecc	2521580	63	16	boe 4, fred 6, ons 6
   2026-09-18	aa4d3ea0e9ba20f2934f06e81683b8a1a0d8d1a4	2521580	63	16	boe 4, fred 6, ons 6
   2026-09-21	809b9e1a6498c295a3c5f49cd1586521f8c4c053	2521580	63	16	boe 4, fred 6, ons 6
   2026-09-22	10f29380ad332ff20d706ad54d8a3acbeb93edfa	2521580	70	18	boe 4, fred 6, ons 8
   2026-09-23	60decda9027a82800dc97f9ed7afd2f3fe2b4e15	2521580	70	18	boe 4, fred 6, ons 8
   2026-09-24	141ee2f81c9ee18130fef67743bceb7915f928cb	2521580	70	18	boe 4, fred 6, ons 8
   2026-09-25	a20102d3b9ebb68ca082ed5e3b16149747e292ce	2521580	70	18	boe 4, fred 6, ons 8
   2026-09-28	399ee7782dc16b181c97e8c39266c6a9527cc583	2521580	70	18	boe 4, fred 6, ons 8
   2026-09-29	612329c0e9313aff242ef11675b9fc3387e0e1b7	2521580	70	18	boe 4, fred 6, ons 8
   2026-09-30	2a3dec3a24712b3698f33124342a84c3b48ab273	2521580	76	20	boe 4, fred 6, ons 10
   2026-10-01	227d8066838051b93f0ff0aab5197dcc7a4f2e75	2521580	76	20	boe 4, fred 6, ons 10
   2026-10-02	9d051f1d48af5d32244e2584e053499ed63843e8	2521580	76	20	boe 4, fred 6, ons 10
   2026-10-05	bbb800b1dd13540c5e447997a62f9495bcda512c	2521580	76	20	boe 4, fred 6, ons 10
   2026-10-06	bf327e00d6fcafcd1debf907aa98b049575094e6	2521580	76	20	boe 4, fred 6, ons 10
   2026-10-07	0219df119a6e4e834978ee31eb4495ee90b710dc	2521580	76	20	boe 4, fred 6, ons 10
   2026-10-08	f6266a3582393167730ae8b41d5d590e152fca70	2521580	76	20	boe 4, fred 6, ons 10
   2026-10-09	849883d0c0f0ac8ec49728a0a133d98130bcfd2a	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-12	4f401e040890cea3cc36f3bbc848f8551a553f1c	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-13	00a045b5a757cd81b414e36b3720e2b518fb0be6	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-14	03d9f5e72bfab3ae64db92a89467efa79f008215	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-15	c855a5ed07058ebb80d3e3a98e52075bd5b9304e	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-16	13e591cec9d4211e7dd03aff74c1beeadef7f9a0	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-19	757113c725e2bae32596898178ab9cb120161fce	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-20	c4b41eef9d292638c0565d4176ec7f826ee545ce	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-21	d654716946ad7902569a8b934d6f2d721077f68c	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-22	adcb65d2511810970a3360b8ae95bcdda46cb4fa	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-23	f0591b83f3dca99a037ea1843414bf388e64886a	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-26	a66b9828130067d6c1b2041bf8a2ff36840f507a	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-27	673c9d87a721f19ebe5c319c905520b91a6317df	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-28	6bdbeba2b5d5601a3e705babad74ae19d2bd5a49	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-29	68b616c217f91218f88bc0cb3786f853226add12	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-10-30	a4be076542a6809209893c4e61c2964755bb73b4	ed47a93	76	20	boe 4, fred 6, ons 10
   2026-11-02	d7e265d2f2f1e49e80c0e9ac13e731a3c5729427	153cf41	76	20	boe 4, fred 6, ons 10
   2026-11-03	9495894464372929ff195cf9bd4e92fb66d94c56	153cf41	76	20	boe 4, fred 6, ons 10
   2026-11-04	a310d213619d8eae4ce17d1186b4f46ef3c78d34	153cf41	76	20	boe 4, fred 6, ons 10
   2026-11-05	18442eafd6cd13355a2a3dc6e0321f2855fb6e26	153cf41	76	20	boe 4, fred 6, ons 10

   "Files changed" covers the regenerated MANIFEST, latest/, notes and pages as well. Only the data/vintages/**/<date>.json files are the evidence.

   Ingesting them (option 1) — #221's refresh.py does not do it already

   Two things in data/refresh.py (#221) prevent it:

   1. --carry-from BRANCH copies the vintage files in that branch's tree (tracked_vintages(origin/BRANCH)). The archive branch's tree is main's, so it would carry nothing. The bot commits are separate trees, not one accumulated tree.
   2. carry_plan skips any file dated at or before main's newest snapshot of that series, which protects the "each file differs from the one before" invariant. When Refresh official data vintages #195 merges, main's newest is 2026-11-05, so after that every 27 Aug–4 Nov file would be skipped by design.

   So recovery has to be a one-off step, and it has to land before or together with #195:

   for sha in <oldest → newest>:
       git diff-tree -r --no-commit-id --name-only --diff-filter=A $sha -- data/vintages
       for each path: git show $sha:$path  → keep only if not fetch.same_data(previous kept, this)
   then rebuild MANIFEST/latest (refresh.py notes→pages→gates) and run the append-only gate
   

   The dedupe matters: commit k and k+1 often hold the same data with a new date, because each was measured against main and not against the day before. The kept files are byte-for-byte what the fetcher wrote (fetched_utc intact). The simplest way to do this is to fold it into #195's branch, adding the deduped files before the 11-05 set and committing them with each source SHA in the message. That is still your decision from the issue body, and it waits on #195 / #200.
2. vahid-ahmadi commented on Nov 6, 2026

   

   vahid-ahmadi

   on Nov 6, 2026

   ContributorAuthor

   More actions

   Recovery tool: draft #228 (stacked on #221)

   data/recover_orphaned_snapshots.py reads the archive branch. It takes each orphan's added vintage files and drops repeat days with fetch.same_data, keeping the first day each release was seen. --apply writes the original bytes (fetched_utc intact). It is a dry run by default and idempotent. It adds no data by itself.

   Dry run on the real archive: 50 commits, 818 files.

   • Before Refresh official data vintages #195: 270 files to add, 548 repeat days dropped.
   • After Refresh official data vintages #195: 265 to add, 20 already present. 15 of Refresh official data vintages #195's 5 Nov files would become permanent repeats of earlier releases, and CPI 2026Q3 / GDP 2026Q2 would be recorded under vintage 11-05 instead of 10-21 / 09-30.

   Recommendation: apply it before #195. The steps are in #228. Nothing has been applied to any branch.
3. vahid-ahmadi

   added

   data

   questionFurther information is requestedFurther information is requested

   on Nov 6, 2026