Consolidate reports-server backlog: partner fixes, test suite + CI, Swapter/NYM/Revolut plugins, v2 dashboard - #232
Open
j0ntz wants to merge 27 commits into
Open
Consolidate reports-server backlog: partner fixes, test suite + CI, Swapter/NYM/Revolut plugins, v2 dashboard#232j0ntz wants to merge 27 commits into
j0ntz wants to merge 27 commits into
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
When an asset has a contract address it is a token, but several plugins silently fell back to a native (tokenId: null) mapping when the token could not be resolved. That prices the token with the chain's gas-token rate and overcounts volume whenever the token is worth less than the gas token. - nexchange: throw when a contract-bearing asset is on a chain whose tokenType is missing, instead of returning tokenId: null. - changenow: drop the try/catch around createTokenId that swallowed failures and returned tokenId: null. - rango: drop the per-tx try/catch that logged and continued, silently dropping any transaction whose asset could not be resolved. All three plugins paginate oldest-to-newest and persist progress on throw, so a failing order halts and is retried next run rather than being mispriced or silently dropped. Also add the SUI and MONAD chain mappings to rango: these were previously dropped silently and would now halt the plugin. Verified by reprocessing the last three months of orders (nexchange 22.7k, changenow 61.9k, rango 2.9k) with zero processing failures. Co-authored-by: Cursor <cursoragent@cursor.com>
Some older LetsExchange transactions return null network fields for unambiguous native assets (e.g. ETH, BTC, XRP), causing processing to throw "Missing network" and stalling the query. Add a currency-to-network fallback for 1:1 native tickers so these transactions process correctly. Co-authored-by: Cursor <cursoragent@cursor.com>
These mappings were causing the query engine to halt and stop scanning forward for the affected partners (fail-closed design), blocking collection of newer transactions until the mapping was added. - SideShift: map SWARMS on Solana to its mint address (delisted from the SideShift coins API, so added to DELISTED_COINS). - Rango: map the SONIC blockchain to the `sonic` Edge pluginId. - Banxa: map the "Primer Paypal Pay" and "Primer Google Pay" payment types to paypal and googlepay respectively. Co-authored-by: Cursor <cursoragent@cursor.com>
The currency cache was built from the `currencies?active=true` endpoint. When ChangeNow deactivates an asset (e.g. DASH), it disappears from the active list while historical transactions still reference it. The lookup then misses and the plugin halts fail-closed, stalling all ChangeNow transaction collection. Fetch the full currency list (omit `active=true`) so previously-listed assets continue to resolve for historical transactions. Co-authored-by: Cursor <cursoragent@cursor.com>
Banxa removed ZEC from the v2 crypto catalog, so historical ZEC orders abort the partner query and stall ingestion past July 8. Co-authored-by: Cursor <cursoragent@cursor.com>
Not private enough and is scrapable. Co-authored-by: Cursor <cursoragent@cursor.com>
Unblocks Banxa query progress stuck since Jul 24 on unrecognized KLARNA Checkout payment methods. Co-authored-by: Cursor <cursoragent@cursor.com>
The suite imported from ../lib/util, a gitignored build artifact that only exists after a build, so a clean checkout could not run any test. Point it at the source module instead. The analytics fixtures also still used the old pluginId key, which the current cleaner rejects.
The suite was not run by CI, which is how it stayed broken. The job installs with --ignore-scripts and generates the gitignored clientConfig.json before running mocha.
Queries Swapter's tool-history reporting endpoint and maps each order to a StandardTx. Pages are buffered so a mid-page failure retries idempotently, and progress only advances once the full walk completes.
Queries NYM's partner reporting endpoint and converts its native-unit amounts to major units using the live currencies list. The report timestamp keys off createdDate because completedDate is null even on settled orders. processNymTx follows the uniform (rawTx, pluginParams) processor contract, and an unparseable order fails its page rather than being skipped: skipping would let the walk complete and advance progress past an order the next run's lookback can no longer reach.
Queries Revolut Ramp's orders endpoint, authenticated with the X-API-KEY header that the GUI's existing ramp key uses. Two shapes of that API are easy to get wrong and are handled explicitly: the start/end bounds are date-only, and the response is a bare array paged by skip/limit rather than a cursor envelope. An order id is not unique in the response. Revolut returns one row per payment attempt, so the same id arrives both COMPLETED and FAILED. Since orderId keys the StandardTx document, attempts are collapsed to one winner per id, settled beating unsettled, before anything is emitted. Native assets resolve to their Edge chain with a null tokenId. Tokens resolve their chain but leave tokenId undefined: Revolut reports no contract address, and a guessed one would mis-price the asset.
Adds an isolated /v2/config route returning per-provider rev-share rates and fiat/swap classification. No v1 route is modified. The rates live on the app doc in reports_apps, as an optional revShareRate beside each partner's apiKeys. The rate is a property of the app-partner deal, so it is per app AND per partner, and it sits with the credentials that define the relationship: onboarding a partner is one doc edit, with no separate rates map to forget. The rates are commercial terms and this repo is public, so they are never committed or placed in config.json. A partner without one contributes 0 estimated revenue.
Ports the redesign prototype into src/demoV2 as a standalone dashboard served at /v2/, built into dist/v2 by a separate parcel invocation so v1's entry, bundle, and output are untouched. The prototype's sample-data layer is replaced by the real /v1 API (getAppId auth, getPluginIds, one analytics POST for all providers) plus /v2/config for rev-share and provider types. A bad or missing apiKey now redirects to a key-entry screen instead of hanging: getAppId returns plain text on 400, so the response is checked before parsing and fetch errors are surfaced.
Some partner APIs report Edge's actual fee per order; estimating that same number as volume times a configured rate discards a fact we already hold. This adds the pair StandardTx.revenueUsd / revenueSource: the actual USD figure, stored at ingest as a fact about the order, and the vocabulary that lets downstream consumers tell a reported figure from a derived one. Revolut is the first reporter: fees_partner_currency.partner_fee, pre-converted to USD by Revolut, taken only on settled orders (an unsettled attempt's fee is not revenue) and only when the settlement currency is USD (a conversion this plugin cannot do honestly). The cache engine sums reported revenue into the analytics buckets, and the v2 dashboard uses it directly wherever a provider reports, marked with a check in the provider table. Providers that report nothing keep the estimate, computed at read time from the app doc's per-partner revShareRate, so correcting a rate fixes history immediately while reported figures stay immutable. Pair rows attribute bucket revenue proportionally to volume share, the same rule the existing tx attribution uses. The revenue keys are optional at the type level rather than spelled out as explicit undefineds across every partner plugin; the cleaner still validates them when present. Cache docs written before this field lack it until the next cache rebuild, which fills history from stored txs with no partner re-query.
The v2 config route carried its own hand-typed fiat/swap table, and it had drifted from the v1 registry it was copied from. It keyed Ionia gift cards, Fox Exchange and NYM by filename or lowercase rather than by their real pluginIds, and omitted banxa2, banxa3 and gebo entirely. Unknown pluginIds default to 'swap' on the client, so all four fiat providers among those rendered under the swap filter with a swap badge. Projecting the table from src/demo/partners.ts removes the duplicate rather than correcting it in place, so the two cannot drift again. bridgeless is kept as an explicit extra: it is registered in edge-exchange-plugins but has never reported a transaction here, so it has no v1 registry entry. This cannot be derived from the data instead. StandardTx.exchangeType exists but is optional and postdates most ingested history, so it is absent from nearly every stored transaction, and PartnerPlugin carries no type at all.
nexchange, Simplex and Bridgeless all lack a working reporting plugin, and none of them is blocked on work this repo can do alone. Record what each one needs, with the live evidence behind it, so the investigation is not repeated. nexchange is credential-scoped: its key authenticates but returns 403 on the reporting resource. Simplex needs a server-side key that only their support issues, and the shipped plugin also targets a retired host and the wrong auth header. Bridgeless has no key at all, but its public API carries no timestamp, no referral filter and no resumable cursor, which is what makes a plugin infeasible rather than merely expensive.
Two defects in the ported plugin, both of which only surface against live data. The nullable response fields used asOptional, which supplies its fallback when a key is missing but still throws when the key is present with an unexpected type. One odd row therefore aborts the whole page, exhausts the retries, and stalls the window, so the same bad row is re-hit every cycle and the plugin never makes progress. asMaybe degrades that row instead. A zero contract address (0x000...0) denotes the chain's native gas asset, not a token, but only a missing or empty address was treated as native. createTokenId would otherwise mint a non-null tokenId for the gas asset and mis-route its rates and volume. Banxa and Moonpay special-case the zero address the same way. Also corrects the blocked-provider doc, which described nexchange as deliberately absent from this repo. That was true of the split branches; this branch carries one copy, so there is nothing left to deduplicate. The credential blocker is unchanged.
j0ntz
marked this pull request as ready for review
August 10, 2026 22:21
j0ntz
force-pushed
the
jon/reports-backlog
branch
from
August 10, 2026 22:21
42d05c3 to
1912adf
Compare
This was referenced Aug 10, 2026
Closed
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Three review findings, each a real defect. Partner-reported revenue never reached CouchDB. checkUpdateTx compares an explicit field list, and revenueUsd/revenueSource were not on it, so a re-poll that fills in a fee without touching an already-tracked field looked unchanged and the write was skipped. Revenue arrives late by nature: a partner can settle the fee after the order row already exists. The v2 dashboard built TODAY from the browser's local calendar day while TODAY_DAY_INDEX and every bucket index are computed in UTC, so anywhere but UTC a day's volume and revenue landed in the wrong slot and the analytics fetch window was misaligned by up to a day. The comment claimed UTC; now the code matches it. The dashboard also left a ?apiKey= seed sitting in the address bar, where it persists in history, copied links and referrers. It is now stripped via replaceState once read. The key still travels as a query param on the API calls themselves, because that is the /v1 auth contract and not something this branch can change unilaterally.
Moving TODAY to UTC midnight fixed the bucket indexes but left the presentation layer reading those UTC timestamps with local getters, which is worse than the original inconsistency because the two halves now disagree. West of UTC a UTC-midnight date renders as the previous local day, so day labels and tooltips shifted by one, and month rebucketing in provBuckets and pairBuckets folded the first UTC day of each month into the previous month on the 12m and 24m charts. fmtDate, both rebucketing sites and the tooltip year now read UTC. The cleaner-failure logs in nym and revolut serialized the entire rawTx, which carries counterparty addresses and transaction ids, into centralized logs where anyone with log access could recover them. Triage needs the record's id and the field names it actually arrived with, which is what identifies shape drift; the full record stays retrievable from Couch by that id.
Every ingestion loop terminated only on a signal from the partner: a null cursor, a short page, or a reported total. That makes a partner endpoint able to decide how long our worker runs, so a stuck cursor or a page that never shortens would spin indefinitely while the in-memory batch kept growing. Each loop now runs under a hard page cap, and hitting it is logged rather than silent. The cap changes nothing about correctness: NYM, Revolut and Swapter only advance their saved progress once the walk completes, so a capped run re-queries the remainder next cycle exactly like the retry-exhaustion path. nexchange paginates oldest to newest and advances progress per processed order, so a capped run keeps the work it did and resumes from there. xgram shares the pattern and is left alone here; it predates this branch.
asValue's type parameter was left to inference, which widened it to string, so StandardTx and DbTx had to be hand-relaxed the same way and every consumer lost the compile-time guarantee: only the runtime cleaner would reject a bad value, after it may already have been persisted. Pinning the literal tuple restores the union everywhere, and the optional-key relaxation stays as it was. Also removes em-dashes from the new comments and docs, per the repo convention that committed code and documentation use a comma, colon or parentheses instead.
ChangeNow, Rango and Xgram all let a per-transaction failure escape the processing loop. That looks like the safe choice, and the comments said so, but it stalls the partner outright: the run halts, progress saves just short of the bad row, and every later poll re-fetches the same range, hits the same row, and dies there. Nothing newer is ever recorded, silently, until someone reads the logs. Xgram was worse again, because its loop sits outside the try/catch that guards the fetch, so the throw rejected queryXgram entirely and discarded every order already processed earlier in the same run. The opposite choice is not available either: emitting the row anyway prices a token with the chain's gas-token rate, which is the failure changenow.ts and nexchange.ts both carry comments forbidding. So the row is quarantined. It is dropped, never emitted with wrong data, and reported at error level with a per-run count so a recurring mapping gap shows up as a number rather than as scattered lines. Ingestion continues past it. Xgram also gains the page cap its four sibling plugins already have.
resolveNetworkCode returned null for two situations that mean different things: an order predating the API's network fields, which is an expected gap, and a recent order whose currency is not in the ~45-entry fallback map, which is a mapping hole. Both were absorbed identically, so the second saved a StandardTx with chainPluginId, evmChainId and tokenId permanently undefined and the asset priced by currency code alone. No error, no retry, nothing in the logs. That is the exact failure changenow.ts and nexchange.ts carry explicit comments forbidding. The two cases are now distinguished, and the unresolvable one is logged at error level naming the currency and the map it needs adding to. The order is still recorded, since dropping it would lose the volume outright; the gap is simply no longer invisible.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit df0f788. Configure here.
Three problems with the quarantine as first written. ChangeNow's guard wrapped processChangeNowTx, which loads the shared currency cache internally, so a currencies-endpoint outage stopped looking like an infrastructure failure and started looking like an unmappable order. Every row would be skipped while the offset advanced, and with no page cap a transient outage could walk the entire window recording nothing. The cache is now loaded once per run outside the guard, where a failure aborts as it did before, and ChangeNow gains the page cap its siblings already have. The skip logs identified nothing, so an operator reading them could not tell which rows were dropped once the watermark moved past them. They now carry the record id and its field names, via the describeRawTx helper that nym and revolut already used, lifted into src/util.ts and shared by all five plugins. The LetsExchange message told operators to add any unresolved currency to LETSEXCHANGE_CURRENCY_TO_DEFAULT_NETWORK. That map is native-ticker only by design, so following it for USDT or USDC would invent a default network and mis-attribute the asset. The message now distinguishes the single-chain case, which the map fixes, from the multi-chain case, which is missing API data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CHANGELOG
Does this branch warrant an entry to the CHANGELOG?
Dependencies
noneDescription
Consolidates every open strand of edge-reports-server work into one branch, per
the umbrella task
reports-server: Umbrella.
jon/reports-backlogis what has been running ondeploy, so this is largely arequest to make
mastermatch what production already serves, plus two fixesfound while assembling it.
Supersedes #228,
#230 and
#231, all closed in
favour of this one. Their content is carried here in full: the branch was
verified file-by-file against each of them before they were closed.
What is in it
Partner fixes (
3dce03f..1b95c0b) — nexchange, Xgram, ChangeNow delistedassets, LetsExchange native-network inference, partner asset/payment mappings,
Banxa Klarna and the ZEC fallback, the getTxInfo endpoint disable, and throwing
rather than pricing tokens as native gas tokens.
Test suite and CI (
397ad8a,e2a6e17) —test/util.test.tsimported from../lib/util, a gitignored build artifact, so a clean checkout could not run asingle test. Repointed at the source module, stale fixture keys renamed, and a CI
job added so the suite cannot silently break again.
Three reporting plugins (
0748c28,8955457,1df0e59) — Swapter, NYM andRevolut, each driven against the live partner API before shipping. Revolut had
been recorded as credential-blocked; it was pointed at a host and path that do
not exist, and the real Ramp API answers with the key already in
env.json.Its orders endpoint returns one row per payment attempt, so attempts are
collapsed per
orderIdbefore anything is emitted.v2 dashboard (
e429c6e,42dd971,ee96687,ede909a) — an isolateddashboard at
/v2/, with v1 untouched. Revenue is a first-class metric: where apartner reports Edge's actual fee it is stored on the transaction as
revenueUsd/revenueSourceand summed through the analytics cache, andelsewhere it is estimated at read time from a per-partner
revShareRateon theapp doc. Rates are never committed, since they are commercial terms and this repo
is public.
Blocked-provider record (
b921135) —docs/blocked-partner-reporting.md,recording what nexchange, Simplex and Bridgeless each need before they can work,
with the live request and exact response behind every claim. Tracked separately
at
its own task.
Review fixes (new here, from the reviewer bots and an independent multi-agent
review) — partner-reported revenue never reached CouchDB, because
checkUpdateTxcompares an explicit field list thatrevenueUsd/revenueSourcewere missing from, and revenue arrives late by nature. The v2 dashboard built
TODAYfrom the local calendar day while every bucket index is UTC, so away fromUTC a day's figures landed in the wrong slot; finishing that migration also took
fmtDate, both month-rebucketing sites and the tooltip year. A?apiKey=seedis now stripped from the address bar rather than left in history and copied
links. Cleaner-failure logs no longer serialize the whole
rawTx, which carriedcounterparty addresses and txids into logs. Every partner pagination loop now
runs under a hard page cap, so a stuck cursor or a page that never shortens can
no longer let a partner endpoint decide how long the worker runs. And
revenueSourcekeeps its literal union:asValue's type parameter was inferredas
string[], quietly costing every consumer the compile-time guarantee.nexchange hardening (
87c7d73, new here) — two defects that only surfaceagainst live data. The nullable response fields used
asOptional, which suppliesits fallback for a missing key but still throws when the key is present with an
unexpected type, so one odd row aborts the page, exhausts the retries and stalls
the window permanently. And a zero contract address (
0x000…0) denotes thechain's native gas asset, but only a missing or empty address was treated as
native, so
createTokenIdwould mint a tokenId for the gas asset and mis-routeits rates and volume.
Testing
verify-repo.sh --base origin/masterpasses end to end: CHANGELOG, install,prepare, eslint over all 25 changed files, and
npm testat 106 passing.The three reporting plugins were each driven against the live partner API when
they were written, read-only with the GUI's existing credentials and never
against production CouchDB: Revolut 1,162 real orders (1,359 raw rows collapsed,
0 duplicate
orderIds), NYM 47, Swapter 14, every emitted row validated againstthe repo's own
asStandardTxwith zero failures.Not verified here: nexchange cannot be exercised at all until a reporting-scoped
key exists, so its fixes are covered by the type checker, eslint and the suite
rather than by live data. The reasoning behind each is in the commit and in
docs/blocked-partner-reporting.md.Partner ingestion robustness (also new here) — the independent review turned
up four correctness defects in partner code that predates this branch, and all
four are fixed.
ChangeNow, Rango and Xgram each let a per-transaction failure escape the
processing loop. That reads as the safe choice, but it stalls the partner: the
run halts, progress saves just short of the bad row, and every later poll
re-fetches the same range and dies on the same row, so nothing newer is ever
recorded. Xgram was worse again, because its loop sits outside the try/catch
guarding the fetch, so the throw rejected
queryXgramentirely and discardedevery order already processed in that run. Emitting the row anyway is not the
alternative either, since that prices a token with the chain's gas-token rate.
So the row is quarantined: dropped, never emitted with wrong data, and reported
at error level with its id and a per-run count. Ingestion continues past it.
LetsExchange treated two different situations identically, so a recent order
whose network could not be resolved was saved with
chainPluginId,evmChainIdand
tokenIdpermanently undefined and no trace in the logs. The two cases arenow distinguished and the unresolvable one is reported, with guidance that
splits single-chain tickers (add to the fallback map) from multi-chain ones
(missing API data, raise with the partner).
Xgram and ChangeNow also gain the page cap the other plugins have, and
describeRawTxnow lives insrc/util.tsshared by all five plugins ratherthan duplicated in two.