Mục lục

Joy Loyalty — Integrations roadmap (v8.1)

How to read citations and tags


0. One-page summary

The problem

  1. Joy loses integration data when a partner rate-limits it. About 205,000 deliveries were dropped in 18 days: 172,229 Omnisend contact updates, 22,479 Klaviyo profile syncs and 10,404 Omnisend events. These are log lines, one per dropped delivery, not unique customers. Nothing retries them and nothing can replay them.
  2. Dead credentials are called for weeks and nobody is told. Klaviyo had 27,166 failed token refreshes on 44 shops (61,240 log lines). Mailchimp, Gorgias (8 accounts, about 3,000 failed refreshes a day, starting 05/11) and Instagram behave the same way. The merchant still sees "Connected".
  3. A large share of the errors is self-inflicted or false. The admin customer page probes Klaviyo and Omnisend for shops that never connected them (40,714 lines). Outbound merchant webhooks count a plain ok reply as a failure (10,491 lines in an 11-day sample). Joy's own email tries a custom domain on every send for shops that have none (about 18,000 lines a day).
  4. Order webhooks were lost while Shopify was told "success". 344 order webhooks in 18 days could not be queued (303 of them "Task size too large", on 9 shops). Shopify never redelivers them, so those orders may never have earned points.
  5. Secrets and personal data are written to logs. 21,382 Mailchimp lines carry the merchant's authorization header; Recharge orders are logged with billing addresses; Apple Wallet request headers are logged (288 lines); every SMTP send logs the recipient and the subject.

Behind all five is one cause: there is no integration module. Each app is hand-wired across about 30 files, on a message path that cannot retry, with no delivery id, no dead-letter queue, no health state and no success metric (sections 2 and 3). Compute and queue throughput are not the problem.

The work, in five groups

  1. Stop data loss now (before BFCM, by 25/11): remove secrets from logs, stop the self-inflicted probes, switch on three alerts, check the Omnisend consent risk, find the orders that lost points, and have support contact shops with dead credentials.
  2. Fix existing integrations: make each known failure retry, stop, or become visible, app by app (Klaviyo, Omnisend, Mailchimp, Gorgias, Instagram, Drip, Attentive, Flow, outbound webhooks, Joy email, Joy Subscription, Shopify order ingress and Admin throttling).
  3. Build the shared integration infrastructure: an outcome log, a registry, a durable delivery worker, a health service, inbound verifiers, a merchant health UI and credentials loaded by reference, proven on one app before the others move.
  4. New integrations: Flow review templates, the Tapcart secure block, the Loop listing and exchange bonus, and a Postscript beta, each behind its gate.
  5. Partner API and cleanup: the F1 partner API on REST v2, the REST contract test, the REST v1 sunset, partner docs, and removal of dead integration paths.

What changes after the rebuild (why rebuild instead of patching)

Who Today After the rebuild
Merchant Sees "Connected" while the credential has been dead for weeks; lost syncs are invisible (205k drops in 18 days, nobody told). A badge and a Home banner say which app broke and what stopped, with Reconnect, Re-sync and one email (3.6, 3.10).
End customer Order points can silently never arrive (344 orders); stale partner profiles show wrong balances in loyalty emails. Orders are accepted durably before Shopify gets 200 (2.19, 2.20); every profile update is retried until it is delivered or quarantined, inside each partner's replay limit, and is never dropped silently (3.4).
Support Searches Cloud Logging by hand, 18-day retention, no per-shop view. Per-shop diagnostics with last success, 7-day outcomes, plain-language error and replay (3.7, 3.16); tickets labelled per integration (1.10).
Developer A new app is ~32 files, 45 edit sites and 17 switch branches; no test catches a missed site; each app reinvents retry and rate limiting. A new app is one registry entry plus one adapter; retry, rate limit, dedupe, health and metrics come from the platform; contract tests catch drift (3.2, 3.4, 3.13).
Business SMS partners such as Postscript cannot be added safely (a duplicate is a second paid SMS); there is no partner API. F1 lets Loop, Tapcart and Okendo build on Joy (5.1); Postscript ships on the delivery contract (4.6); Joy leads where no competitor is (4.7).

The core change on one delivery. Today, when Omnisend answers 429 the handler sleeps, retries inside the same request, returns undefined, and the dispatcher returns 'ok': the update is gone after one log line (172,229 times in 18 days). After the rebuild, the producer writes a pending delivery with a stable deliveryId (a UUID) before acknowledging. The worker admits it within the shop's budget, which is set well below the partner's published limit because that limit is usually shared with the merchant's other apps. On a 429 the worker creates a new Cloud Task scheduled after the wait that this partner's adapter reads, without counting it as an attempt, and answers 2xx to Cloud Tasks. Omnisend sends no Retry-After header: the adapter reads retryAfter from the 429 body when it is there and otherwise backs off with jitter. The deliveryId goes into the partner's dedupe field only where the partner documents one that works (Klaviyo unique_id, LINE X-Line-Retry-Key). Omnisend has none for events that trigger automations, so an Omnisend event whose send timed out is quarantined for operator replay instead of being resent blindly. A 401 follows the same idea: auth_broken stops the calls, keeps the tokens and tells the merchant. A 403 for a missing scope or plan is not a 401: it blocks one capability and leaves the connection alone.

How each failure is fixed is in the "What" line of every item in section 6.

1. Integrations in use

One row per integration that has runtime code, ordered by merchants connected. Merchants connected = distinct shops with an active connection whose Joy install is still live (shops.isInstalled == true, out of 20,835 installed shops); incl. uninstalled adds shops that have since removed Joy. Counted read-only from prod Firestore on 20/11/2026, aggregates only: integration-usage-counts.md. Flow, Joy Subscription and Wallet come from shop-level flags; the rest from the integrations collection (isConnected == true). "not tracked" = no connection record exists to count (Joy-provided APIs, Smax, Chatty, Joy Wishlist, Klaviyo Reviews, Facebook Page, order webhooks). Installed is an upper bound: frozen or closed stores keep isInstalled: true. The Integrations page also shows 17 link-only cards (for example Okendo, REVIEWS.io, Skio); they have no runtime connection and are not listed here.

App Category Merchants connected incl. uninstalled How it connects (direction, transport)
Shopify Flow Shopify platform 1,456 2,097 Outbound: Flow triggers through the Admin API. Inbound: HMAC-signed Flow action calls over HTTP
Judge.me Reviews 1,114 1,944 Inbound: unsigned review webhooks over HTTP. Outbound pull: a daily review backfill (OAuth)
Klaviyo Email/SMS marketing 790 1,029 Outbound: loyalty events, profile changes and bulk re-sync, Pub/Sub then Klaviyo REST API (OAuth or a private key). Inbound: see Klaviyo Reviews
Joy Subscription Subscriptions (Avada app) 428 631 Inbound: signed webhook over HTTP. Outbound: plan and install calls over HTTP
Air Reviews Reviews 332 517 Inbound: review webhook over HTTP with an HMAC header
Instagram Social 239 282 Inbound: signed Meta webhook over HTTP, then Pub/Sub. Outbound: Graph API calls (OAuth)
Google Wallet Wallet 178 214 Outbound: pass object updates, Pub/Sub then Google Wallet API. Inbound: save callback over HTTP
PushOwl Web push 154 198 Outbound: push notifications for loyalty events, Pub/Sub then PushOwl API (partner key and a per-shop token)
Omnisend Email/SMS marketing 149 202 Outbound: events, contact updates and bulk batches, Pub/Sub then Omnisend REST API (OAuth)
Apple Wallet Wallet 139 169 Outbound: APNs push when a customer changes. Inbound: PassKit web service and pass downloads over HTTP
Yotpo Reviews 97 156 Inbound: unsigned review webhook over HTTP
Mailchimp Email marketing 75 96 Outbound: audience member updates, Pub/Sub then Mailchimp REST API (OAuth); bulk batches polled by a 5-minute cron
Gorgias Helpdesk 58 61 Outbound: customer data for the helpdesk widget, Pub/Sub then Gorgias REST API (OAuth)
Fera Reviews 50 88 Inbound: unsigned review webhook over HTTP
Recharge Subscriptions 23 28 Inbound: unauthenticated order webhook over HTTP. Outbound: discount sync through a Cloud Task (API token)
Sendlane Email marketing 13 13 Outbound: bulk list sync only, on connect or manual re-sync (API token)
Loox Reviews 10 10 Inbound: review webhook over HTTP, HMAC-signed when the shop has a key
LINE Messaging 5 5 Outbound: push messages, Pub/Sub then LINE Messaging API. Inbound: signed Messaging API webhook
Review Rocket Reviews 4 4 Inbound: unsigned review webhook over HTTP
Drip Email marketing 3 3 Outbound: subscriber fields, Pub/Sub then Drip REST API (OAuth); bulk continued by an hourly cron
Intercom Helpdesk 2 2 Outbound: contact attributes, Pub/Sub then Intercom REST API (OAuth or a personal access token)
Attentive SMS marketing 0 1 Outbound: events, custom attributes and bulk sync, Pub/Sub then Attentive REST API (OAuth)
Stamped Reviews 0 (no connected doc) 0 Connect-time credential check only; no review ingestion path exists
Smax Messaging not tracked — Outbound: point and tier changes, Cloud Task then Smax API (one global token for all shops)
Chatty Helpdesk / chat (Avada app) not tracked — Both: Chatty calls /integrate/oauth/connect with one shared constant key; Joy calls Chatty with the same key
Klaviyo Reviews Reviews not tracked — Inbound: review webhook over HTTP, HMAC-signed when a secret is stored
Joy Wishlist Wishlist (Avada app) not tracked — Inbound: events over HTTP (JWT). Outbound: wishlist API calls; bulk chunks through a Cloud Task
Facebook Page Social not tracked — Inbound: signed Meta webhook over HTTP, then Pub/Sub. Outbound: Graph API calls (OAuth)
Shopify order webhooks Shopify platform not tracked — Inbound: HMAC-signed webhooks over HTTP, then one Cloud Task per order
Outbound merchant webhooks Joy-provided API not tracked — Outbound: HMAC-signed HTTP calls to the merchant's URL, Pub/Sub for activity topics, Cloud Tasks for program topics
Email sending (Mailgun custom domain, merchant SMTP, default SMTP) Email delivery not tracked — Outbound: every notification, Pub/Sub then SMTP or Mailgun
REST API v2 (and v1) Joy-provided API not tracked — Inbound: merchants and partners call Joy over HTTPS with an app key and a secret key
Headless API Joy-provided API not tracked — Inbound: headless storefronts and mobile apps call Joy over HTTPS with a headless token
MCP Joy-provided API not tracked — Inbound: AI clients call Joy through the MCP server (OAuth 2.1), which calls REST

Two findings from the count: (1) the two connection sources disagree (Klaviyo 790 by integration doc vs 761 by shop flag; Judge.me 1,114 vs 1,216), which is the "four answers to connected?" problem of §2.1, now measured; (2) many connections sit on shops that uninstalled Joy (Judge.me 830, Klaviyo 239), consistent with calls still made on dead credentials (§4.1).

Source: research-infra.md section 8 (every row there is listed here), plus Chatty from its section 1.5.


2. Current architecture (from code at HEAD)

2.1 How an integration is wired today

There is no integration module. One app is spread over about 30 files. Attentive, the closest template for a new outbound app, touches about 32 files and 45 edit sites, 17 of them branches in switch or if chains (a count of today's touch points, not a commit diff).

Constants. The app id lives with the loyalty-program constants, for example ATTENTIVE_APP at F/const/programs/integrationPrograms.js:11. The event list and the message and job types are separate constants (F/const/integrations.js:338, F/const/integrations.js:376). Id values use four casings and one app has two ids, so nothing can be derived from the id.

Switch chains. Adding an app means adding a branch in each of these places:

No test fails when one of these sites is missed.

Connection flags. "Is this app connected?" has four answers that can disagree:

  1. the shop document flag statusConnected* (19 names), used as a cheap gate by the fan-out (F/services/triggerIntegrationAppService.js:453-476);
  2. the integration document field isConnected, shown by the admin list (A/pages/Integration/Integration.js:62-79);
  3. the admin's fallback map, which reads seven flags that the backend never writes, such as statusConnectedDripApp and statusConnectedJudgeMeApp (A/const/integrationStatusFields.js:32-42); the backend writes statusConnectedDrip and statusConnectedJudgeApp;
  4. the AI tool's map, which reads three of the same unwritten flags (F/services/ai/tools/integrationToolService.js:183-262).

The seven writers of connection state. At least seven code paths change whether an app counts as connected, and they do not agree on which fields to write:

  1. the connect callbacks, one per OAuth app (for example F/controllers/integrate/klaviyoController.js:23, F/controllers/integrationController.js:688);
  2. the merchant's disconnect (F/controllers/integrationController.js:562-601, F/repositories/integrationsRepository.js:699-790);
  3. the repository's side effects after any save, which set the shop flag (F/repositories/integrationsRepository.js:806-897);
  4. the Klaviyo auto-disconnect on a 401, which also revokes the grant at Klaviyo (F/services/klaviyoService.js:349-354 calls F/repositories/integrationsRepository.js:899-933; the revoke is at :910-925);
  5. the Intercom auth failure, which sets isConnected: false and an error (F/services/intercomService.js:115-121);
  6. the Drip failure path, which sets the shop flag to false on any error, including a transient one (F/services/dripService.js:371-378);
  7. the generic settings update, which derives the shop's event-enable flags (F/controllers/integrationController.js:457-550).

A read path also changes state: the admin's live Klaviyo check flips isConnected only in the HTTP response (F/controllers/integrationController.js:322-341).

2.2 Outbound path (Joy to partner)

  1. Business code calls sendEventToIntegrations for an event (about 40 call sites) or updateCustomerToIntegrations for a profile change (about 14 call sites).
  2. That function publishes one Pub/Sub message that contains the shop, the customer and the full integration documents with their tokens (F/services/triggerIntegrationAppService.js:83-98). If the publish fails, the error is logged and the event is lost at the source (F/services/triggerIntegrationAppService.js:99-103).
  3. The message goes to backgroundHandlingLight (events) or backgroundHandlingSmall (profiles). Both subscriptions run with retry: false (F/entries/pubsub-background.js:39-47).
  4. The dispatcher runs the case and returns 'ok' whatever happened (F/handlers/pubsub/subscribeBackgroundHandling.js:1717-1727).
  5. The fan-out runs every app in one Promise.all and swallows per-app errors (F/services/triggerIntegrationAppService.js:209-213, F/services/triggerIntegrationAppService.js:367-371).
  6. Each app service calls the partner. What happens on a 429 or an error differs per app (section 3.2). Klaviyo profiles, Intercom and LINE re-enqueue themselves on the shared Cloud Tasks queue through three hand-written helpers (F/helpers/klaviyoTaskQueue.js:27-50, F/helpers/intercomTaskQueue.js:25-36, F/helpers/lineTaskQueue.js:21-40).

Profile changes have two more drop points: a change is skipped while a bulk tag, metafield or tier-launch run is active (F/services/triggerIntegrationAppService.js:326-336), and each per-app publisher skips while the integration's bulk sync is running (F/repositories/customerRepository.js:7236, F/repositories/customerRepository.js:7265, F/repositories/customerRepository.js:7301).

2.3 Inbound path (partner to Joy)

Partner and Shopify webhooks arrive on HTTP functions and are processed inline in the request. Each source has its own verifier, written differently:

Order webhooks are the one inbound flow that is queued: the middleware puts the whole request state (shop document plus order payload) into a Cloud Task (F/middleware/webhook/enqueueWebhookToCloudTasks.js:37-41). If that enqueue fails, the error is logged and Shopify still gets a success answer (F/middleware/webhook/enqueueWebhookToCloudTasks.js:46-58).

2.4 Credentials

2.5 Diagram of today

flowchart LR
  subgraph ADMIN["Admin UI"]
    CARDS["Integrations list and detail pages"]
    CUST["Customer page sync status"]
  end
  subgraph CODE["Per-app wiring, no module"]
    IC["integrationController<br/>4 app switches"]
    IR["integrationsRepository<br/>connect, disconnect, side effects"]
    TIS["triggerIntegrationAppService<br/>fan-out lists and gates"]
  end
  subgraph STATE["State in Firestore"]
    DOC[("integrations doc<br/>isConnected plus tokens in clear")]
    FLAG[("shop flags<br/>statusConnected x19")]
    RL[("integrationRateLimits<br/>3 apps only")]
  end
  subgraph ASYNC["Async path"]
    PS["Pub/Sub backgroundHandling<br/>retry off"]
    DISP["Dispatcher, 212 cases<br/>always returns ok"]
    CT["Cloud Tasks enqueueSubscriberv2<br/>one attempt"]
  end
  BIZ["Loyalty events and customer changes"] --> TIS
  CARDS --> IC
  CUST -->|"live partner call per view"| IC
  IC --> IR
  IR --> DOC
  IR --> FLAG
  TIS -->|"message carries tokens"| PS
  PS --> DISP
  DISP --> SVC["Per-app services<br/>own retry and rate limit"]
  SVC --> RL
  SVC -->|"hand-written re-enqueue"| CT
  CT --> SVC
  SVC --> PARTNERS[("Partner APIs")]
  SVC -->|"error: log one line, drop"| LOST["Lost work"]
  HOOKS["Partner and Shopify webhooks"] --> VER["One verifier per source<br/>4 unsigned, 2 fail-open"]
  VER --> INLINE["Inline handlers<br/>HTTP 200 on failure"]
  VER -->|"orders only"| CT

3. Current infrastructure per integration type

3.1 Shared runtime

Everything below is defined in F/entries/*.js and was re-read at HEAD.

HTTP functions. The integration endpoints set no region, so they run in us-central1.

Pub/Sub topics and retry settings.

Cloud Tasks. One shared queue, enqueueSubscriberv2: 120-second timeout, 1 GiB, retryConfig.maxAttempts: 1, no rateLimits (F/entries/tasks.js:16-23). It carries order webhooks, the Klaviyo, Intercom and LINE re-enqueues, bulk retries, program webhooks and the Recharge discount sync. Because the queue allows one attempt, every retry is a hand-written re-enqueue. A dedicated, rate-limited queue with real retry settings exists for another feature and is the template for integrations (F/const/pointExpiration.js:39-43).

Schedulers. Mailchimp job polling and the Klaviyo and Omnisend import-job checks run inside the 5-minute cron (F/handlers/schedule/frequentOperations.js:14-18); Drip sync runs inside the hourly cron (F/handlers/schedule/hourlyOperations.js:16); the Judge.me backfill runs daily at 01:00 (F/entries/schedule.js:43-44). No cron refreshes OAuth tokens. No cron re-sends failed merchant webhooks: the catch-up job publicSendWebhookForActivitiesMissing.js exists but is not registered. No cron drains Smax batches.

Firestore collections.

Deploy. In CI only staging 12 and production deploy Firestore indexes, so a new queue, collection, index or secret must be provisioned on every staging slot that is used for a test.

3.2 Per integration type: what runs it and what happens on failure

Email and SMS marketing, outbound (Klaviyo, Omnisend, Mailchimp, Attentive, Drip, Sendlane, PushOwl)

Helpdesk and messaging (Gorgias, Intercom, LINE, Smax, Chatty)

Reviews, inbound (Judge.me, Yotpo, Loox, Stamped, Fera, Review Rocket, Air Reviews, Klaviyo Reviews)

Subscriptions and wishlist (Recharge, Joy Subscription, Joy Wishlist)

Social (Instagram, Facebook Page)

Wallet (Apple Wallet, Google Wallet)

Email sending (Mailgun custom domain, merchant SMTP, default SMTP)

Shopify (order ingress, Admin API, Flow)

Joy-provided APIs (REST v2, headless, MCP, outbound merchant webhooks)

3.3 Is today's infrastructure enough?

For scale, yes. For reliability, no. Cloud Functions, Pub/Sub and Cloud Tasks handle far more traffic than 5 to 8 more partners add. The Cloud Tasks limits (500 dispatches per second per queue, 1,000 queues per region by default) are not a problem, but they rule out one queue per shop. What is missing is everything that makes a delivery safe:

Capability Verdict Evidence
Durable queue with retry and backoff Exists, but integrations do not use it Integration topics are retry: false; the shared queue allows one attempt; errors are swallowed (F/services/triggerIntegrationAppService.js:209-213)
Rate limiting that honours each partner's own retry signal Partial Three apps share integrationRateLimits; Omnisend sleeps in the handler; Gorgias limits per instance; no queue-level rateLimits. Only Klaviyo documents a Retry-After header; Omnisend, Attentive, LINE, Postscript, Drip and Mailchimp do not
Idempotency key per delivery Missing Klaviyo's unique_id falls back to a random id (F/services/integrateKlaviyoService.js:453); no integration task has a name
Dead-letter queue and replay Missing for integrations Only the ordered topic has one, attached by a manual script that is not in CI (ROOT/scripts/setup-pubsub-ordering.sh:136)
Auth circuit breaker that tells the merchant Missing except for SMTP Section 3.2; 61,240 Klaviyo refresh-failure lines in 18 days
Success and failure metrics per app Missing Apps log failures as free text and do not log successes
Inbound signature verification Partial Four sources with no check in Joy (two of them, Judge.me and Fera, do sign), two fail-open, one unauthenticated (section 2.3)
Secrets handling Partial Tokens in message payloads and in logs; one constant key in source

The prod logs in section 4 show the cost of each gap.


4. What the prod logs show (18 days)

Method. Cloud Logging entries.list on resource.type="cloud_run_revision" (all regions), 2026-11-02 → 2026-11-19, paginated to exhaustion, read-only with the support service account. The _Default bucket keeps 18 days, so the counts below are the frozen baseline: the raw entries expire, these counts do not.

4.1 Findings by integration

Each row gives what the logs show, the cause type as "trigger → why it persists", the code evidence, and the item in section 6 that fixes it.

Klaviyo

Finding What the logs show Trigger → why it persists Evidence in code Fixed by
Auth probe Unauthorized Klaviyo access 25,438 lines (exact) = 12,719 failed calls, each logged again by its caller; 810 shop ids; 24,246 lines on apiv2/apisav2, 1,192 on enqueuesubscriberv2 (the sync path, not the probe) CODE (admin page probes unconnected shops) → CODE (a read path calls disconnectIntegration) A/pages/Customer/Details/IntegrationStatus.js:46-54; F/helpers/integration/klaviyo/checkKlaviyoSyncStatus.js:15-27; F/services/klaviyoService.js:349-354 R-01
Dead OAuth Failed to refresh token. 61,240 lines (exact) = 27,166 failed refreshes, the rest are callers logging the same failure; 44 shops; top 3 ≈ 40k lines PARTNER/CONFIG (merchant revoked or uninstalled; some may have uninstalled Joy itself) → CODE (invalid_grant branch unreachable, axios throws on 4xx; the branch would also revoke and null tokens) + INFRA (no breaker) F/services/klaviyoService.js:717-758 (revoke at :732-748) R-02, P-05
429 burst 429 6,241 (exact; 5,466 on 11-13). Throttle/re-enqueue lines ≥ 400,000 (capped), 187k from one shop in 3 hours. 22,479 profile syncs dropped after 3 tries (exact; 7 shops, 21,967 from one shop on 11-12/13). 498 bulk batches dropped after their one retry was throttled again, 46 more on a refresh failure. INFRA (mass operation for one shop) → INFRA (the throttle gate re-enqueues and each deferral counts toward MAX_KLAVIYO_RETRIES = 3; the throttle cache is per instance for 30 s). Production-version-dependent: demotion has used bulk collectors with skipKlaviyo since a37aecb835 (07/10), so the drops are on the per-customer klaviyoSync path the collectors do not cover Collectors F/handlers/pubsub/subscribeBackgroundHandling.js:2563-2564, 2638; drop F/handlers/schedule/enqueueHandler.js:104, 544-549; cache F/repositories/integrationRateLimitRepository.js:30-59; bulk drop F/services/integrateKlaviyoService.js:991-1030 P-04b, P-04
Missing scope MISSING_EVENTS_WRITE_SCOPE 1,326 lines (exact) = 663 failed events (two lines each), 3 shops CONFIG (key lacks events:write) → UX (merchant not told). Event capability only: profile sync still works F/services/klaviyoService.js:349-350 P-05 (capability block), P-09

Omnisend

Finding What the logs show Trigger → why it persists Evidence in code Fixed by
Contact 429 error updateOmnisendContact 177,491 (exact): 172,229 rate-limited and dropped, 5,262 other; 63 shops (top 3 = 157k); spikes 11-08 (52k), 11-13 (23k), 11-16 (84k) INFRA (mass updates fan out per contact) → INFRA (retry sleeps inside the handler, then returns undefined) F/services/integrate/syncOmnisendService.js:373-399 P-04, P-04b (after R-18)
Event 429 10,404 events dropped (exact), 4 shops PARTNER limit → INFRA (sendEvent throws on 429, caller logs and returns; no dedupe id sent, and Omnisend's eventID does not dedupe events that trigger automations) F/services/integrate/omnisendService.js:66-80, 163-188; syncOmnisendService.js:509 P-04 slice 1
Probe shop not connect omnisend 15,276 (exact) CODE (same admin probe; cheap, no network) F/services/integrate/syncOmnisendService.js:37 R-01
Consent (code only, no log signal possible) Every contact upsert sends channels: {email: {status: 'nonSubscribed'}}, or sms for phone-only contacts. Whether this downgrades subscribed contacts is unproven. If it does, each sync shrinks merchants' audiences SEC/CODE F/services/integrate/omnisendService.js:221-231 R-18

Mailchimp, Gorgias, Instagram (dead credentials)

App What the logs show Trigger → why it persists Evidence in code Fixed by
Mailchimp Sample 2026-11-10 → 19 (per-day cap): 401 ≈ 12.2k, 403 ≈ 5.3k, 429 ≈ 2.2k across Error checking email in list (18,493 lines) and Error updating contact in Mailchimp (2,889 lines); all 21,382 lines carry the merchant's authorization header PARTNER/CONFIG (revoked key) → INFRA (no breaker); SEC (whole error object logged) F/services/integrate/mailchimpService.js:196 R-03, P-05
Gorgias Sample (cap reached at 42 hours, 18/11 → 19/11): refresh fails with invalid_client on 8 accounts, ≈ 3,000 failed refreshes/day; each failure then throws Cannot destructure property 'accessToken' (≈ 5,900 lines/day; ≈ 8,900 error lines/day in total). The first failure in the retained logs is 2026-11-05 01:46 UTC; there are none on 02/11-04/11 PARTNER/unknown (something changed on 05/11) → CODE (handleRefreshToken returns undefined) + INFRA (no disconnect). Unproven whether all Gorgias shops fail F/services/gorgiasService.js:198-238 R-05, P-05
Instagram Expired or invalidated session: ≈ 6,300 error lines in a 14-day capped sample (05/11 → 19/11), 140-730 lines/day on every day; 1,492 failed token refreshes CONFIG (session expired, or merchant changed password) → INFRA + UX (the refresh is retried on every use of the integration, there is no cron and no reconnect prompt) F/services/instagram/instagramApiService.js:41-47, 103-105 R-14

Reviews and subscriptions

App What the logs show Trigger → why it persists Evidence in code Fixed by
Loox 17 Invalid webhook signature in 18 days Verification works where a key exists (401 returned). Legacy installs without a key skip verification F/middleware/webhook/verifyHookLoox.js:70-84; key created on connected save F/repositories/integrationsRepository.js:80-82 IP-08, P-07
Judge.me ~2.5k review hooks (2,475), no failure clusters Webhook unsigned (IP-07) F/middleware/webhook/verifyHookReviewApp.js:17-57 P-07
Recharge Order logged as JSON with billing address SEC; endpoint unauthenticated (IP-09) F/services/integrate/rechargeService.js:281, 296 (route F/routes/apiHookV1.js:627) R-03, P-07
Joy Subscription 724 Ignoring unhandled topic: subscription_billing_attempts/failure (18 days); 216 HTTP 401 in Joy Sub API responses PRODUCT (topic subscribed but not handled) / CONFIG (401) F/controllers/joySubWebhookController.js:84 R-21

Joy's own outbound channels

Channel What the logs show Trigger → why it persists Evidence in code Fixed by
Outbound merchant webhooks 20,057 send hook error lines in ~11 days (capped sample, 09/11 → 19/11) on webhooksubscriberv2. 10,491 are endpoints that answered the plain body ok; ~920 answered other plain text (skipped, updated, created, …); ~8,600 are error pages or network errors (Internal Server Error 6,678, Service Unavailable, upstream, fetch failed). The HTTP status is not logged, so "2xx" is inferred from the body (a) CODE: a successful 2xx with a non-JSON body becomes status 500 → R-16. (b) PARTNER trigger (merchant endpoint down) → INFRA persists (no retry or catch-up for failed deliveries) → P-11 F/services/webhookService.js:89-95 (result.json() inside the try) R-16, P-11
Joy email, custom domain ~18,000/day (19,921 lines in a 27-hour capped sample, 18/11 → 19/11) Custom domain sending failed, falling back to default SMTP: No verified custom domain found on sentemailnotificationv2 CONFIG trigger (shop has an activated sender but no verified domain) → CODE persists (the gate checks only activatedSender.isActivated, so every send for such a shop reads the domain, fails and logs) Gate F/services/email/sendEmailService.js:129-131; result :209-213; log :162-164, 168-170 R-17
Joy email, SMTP PII Every Message sent via SMTP line logs the subject and the recipient email SEC (log line written on every send) F/services/email/sendEmailService.js:404 R-03
Wallet Apple pass download logs the request headers including authorization: ApplePass … (288 lines in 18 days, on every day). Push-token details are logged (183 lines). SEC; confirms the IP-14 remnant, with a second site found at HEAD Headers F/controllers/loyaltyPassController.js:302-306 (download) and :459-463 (updatable passes); pass/customer dump :345; push token F/services/AppleWallet/ApplePushNotificationService.js:79-82, ~104 R-03 (IP-14 remnant)

Shopify

Surface What the logs show Trigger → why it persists Evidence in code Fixed by
Shopify Flow email is a required field 348 (sample, 17 days) on apishopifyflowv2 CONFIG (workflow sends empty email) → UX (no guidance) + CODE (200 on failure) F/controllers/shopifyFlowController.js:275, 342, 500, 1198-1203 R-15
Shopify Admin API (throttling) Sample (cap reached after ~28 hours, 18/11 14:52 → 19/11 18:36 UTC): graphql error … Throttled 12,322 on 28 shops (one shop = 7,747): 8,397 on apihookv2 (inline in webhook handlers), 2,318 on enqueuesubscriberv2, 1,088 on backgroundhandlingorderedv2. The same failures are logged again as Error when get customers shopify 3,044 (all on apihookv2), Error in handleAddOrRemoveTagCustomer Throttled 889 and Error when remove tags customer … Throttled 820 (both on backgroundhandlingorderedv2). ≈ 10.7k throttled calls/day; the customer reads return empty and the tag operations are logged and dropped INFRA (per-customer Admin calls inline in webhook paths and on the ordered topic, no cost-aware per-shop queue) → INFRA (no retry: the customer read returns {}, and the dispatcher catches the tag error and returns 'ok'; tag sync silently incomplete) Customer read F/services/shopify/shopifyMetaFieldService.js:3453, 3515; tag writes F/services/shopifyService.js:3272-3311, 3775; dispatcher F/handlers/pubsub/subscribeBackgroundHandling.js:1717-1727 R-19
Order webhooks → Cloud Tasks Error enqueueing order webhook to Cloud Tasks: 344 in 18 days (exact, 49 shops): 303 "Task size too large" (9 shops, one shop = 263; 302 orders/updated, 1 orders/create), 32 HTTP 502, 5 HTTP 408, 4 Internal authentication error. The webhook still answers success to Shopify, so Shopify never redelivers: these orders may never have been processed for points CODE (the whole ctx.state, shop document plus order payload, is put in the task, :37-41; Cloud Tasks size limit) → INFRA (enqueue failure is logged and returns success: IP-18) F/middleware/webhook/enqueueWebhookToCloudTasks.js:37-41, 46-58 IP-18, IP-18b, and the affected-orders check

No error clusters

4.2 Root-cause rollup

Exact or capped-sample counts (shares over the displayed counted fingerprints; Mailchimp is a 10-day sample, Flow a 17-day sample; not an all-errors fleet distribution):

Class (by why it persists) Log lines Share Apps Fixed by
INFRA: work dropped after rate limit (no durable retry, no bulk path) 205,112 (172,229 + 22,479 + 10,404) 61.0% Omnisend, Klaviyo P-04, P-04b
CODE + INFRA: dead credential retried, no breaker 61,240 (Klaviyo; 27,166 failed refreshes) 18.2% Klaviyo R-02, P-05, P-09
CODE: self-inflicted probe from the admin 40,714 (25,438 + 15,276; 1,192 of the Klaviyo lines come from the sync path) 12.1% Klaviyo, Omnisend R-01
SEC: secrets/PII in logs 21,382 (Mailchimp sample) 6.4% Mailchimp R-03
Klaviyo 429 lines themselves 6,241 1.9% Klaviyo P-04b
CONFIG → UX: misconfiguration nobody sees 1,674 (1,326 + 348; 663 Klaviyo events) 0.5% Klaviyo, Flow P-05, P-09, R-15
Total 336,363 100%

Sampled or approximate, not in the shares above (different windows):

Class Volume Apps Fixed by
CODE: false failure (2xx with non-JSON body counted as failure) 10,491 in an ~11-day capped sample Outbound merchant webhooks R-16
PARTNER → INFRA: merchant endpoint error page or network error, no retry ~8,600 in the same sample (20,057 − 10,491 − ~920 other plain-text replies) Outbound merchant webhooks P-11
CONFIG → CODE: work attempted on every call though the config can never succeed ~18,000/day (27-hour sample) Joy email (custom domain) R-17
CODE + INFRA: dead credential retried (samples) Mailchimp ~17.4k (401 + 403) / 10 days; Gorgias ~8.9k error lines/day; Instagram ~6.3k / 14 days Mailchimp, Gorgias, Instagram R-05, R-14, P-05
SEC: secrets/PII in logs Wallet auth headers 288 / 18 days; SMTP recipient/subject on every send; Recharge, Omnisend PII Wallet, Joy email, Recharge, Omnisend R-03
INFRA: Shopify Admin throttling, no cost-aware queue ~10.7k throttled calls/day (28-hour sample, 28 shops) Shopify Admin API R-19
CODE → INFRA: order webhook not enqueued, success returned 344 / 18 days (303 "Task size too large") Shopify order webhooks IP-18, IP-18b
Working as intended Loox invalid signatures 17 / 18 days Loox —

The five cause types, read across both tables:

4.3 What the logs say about the whole system

Not verified: which production build ran on 13/11; whether Omnisend downgrades consent; merchant demand for the new apps; the complete Stamped and Wishlist pulls. From the partner documentation, also not verifiable: Sendlane and PushOwl limits and error codes; whether Klaviyo refresh tokens rotate; whether Postscript external_id and Attentive externalEventId dedupe; the retirement date of Omnisend /v5; what Gorgias means by invalid_client; what Tapcart's customerSession carries; Loop's listing process; Review Rocket's webhooks.


5. Target architecture and infrastructure

5.1 Diagram

flowchart LR
  subgraph PRODUCERS["Producers"]
    EV["Loyalty events, customer changes, crons"]
    UI["Admin UI"]
  end
  REG[("Integration registry<br/>browser-safe metadata plus server adapters")]
  EV --> ACC[("Durable acceptance<br/>pending delivery record with a logical deliveryId")]
  ACC --> DW["Delivery worker on a dedicated Cloud Tasks queue<br/>per-shop admission, attempt id differs from deliveryId"]
  SW["Sweeper"] -.->|"repairs failed handoffs"| ACC
  DW --> AD["Outbound adapters<br/>Omnisend, Klaviyo, Postscript<br/>each owns retry timing, payload limit and dedupe field, if the partner has one"]
  AD --> PARTNER[("Partner APIs")]
  DW --> OUT[("Outcome: success, retryable,<br/>or ambiguous to quarantine")]
  IN["Partner webhooks"] --> VER["Per-provider verifier<br/>tenant binding, dual-accept window"]
  VER --> INH["Inbound handlers"]
  AD --> HS["Health service<br/>per app, fenced by installation generation"]
  INH --> HS
  HS --> DOC[("integration doc health<br/>connected, degraded, auth_broken, capability blocks")]
  DOC --> UI
  DOC --> FLAG[("shop flags as a derived cache<br/>plus a repair script")]
  REG -.-> DW
  REG -.-> VER
  REG -.-> HS
  REG -.-> UI
  PAPI["Joy partner API F1 on REST v2<br/>default-deny scopes, quota per shop and partner"] --> EXT["Loop, Tapcart, other partners"]

5.2 The components

Integration registry. One code module declares every app: its id, the legacy ids stored in Firestore, its category, the shop flag it owns, its event list and its credential fields. Browser-safe metadata is kept apart from the server adapters. Each entry names its connection source or says "not trackable" (link-only cards). A drift test fails the build when the registry and the real writers disagree. No stored id, flag, route or payload changes shape; the registry aliases the legacy names.

Delivery worker. One dedicated Cloud Tasks queue carries every outbound delivery. A delivery is built from one envelope: schema version, logical deliveryId, attempt id, installation generation, operation, source event id and time, an immutable payload or a reference to it, expiry and status. The worker:

Eight rules come from the partners' documentation and hold for every adapter:

  1. Retry timing is per adapter. Only Klaviyo documents a Retry-After header on a 429 (an integer number of seconds). Omnisend returns a retryAfter field in the 429 body on some endpoints and no header; its window slides, so there is no reset moment. Intercom returns X-RateLimit-Reset as a Unix time, and not on every 429. Gorgias returns Retry-after. Attentive, Postscript, LINE and Drip document no wait at all. Drip's window is one hour.
  2. Partner-side dedupe exists for two partners only. Klaviyo unique_id dedupes per profile and metric, so the deliveryId must be fixed before the first attempt and the metric name must not change between attempts. LINE X-Line-Retry-Key dedupes for 24 hours. Postscript external_id and Attentive externalEventId exist, but neither partner documents that it dedupes on them. Omnisend eventID is documented as not working for events that trigger automations. Mailchimp, Drip, Intercom, Sendlane and PushOwl have none. For every partner outside the first two, an ambiguous send is quarantined, or the call must be a state update that is safe to repeat (a Mailchimp member upsert by subscriber hash, a Drip subscriber update by email). A state update still needs "latest state wins", because a retried older update can overwrite a newer one.
  3. The deliveryId is a UUID. Omnisend eventID and LINE X-Line-Retry-Key accept nothing else.
  4. Budgets sit well below the published limit. The quota is usually shared with the merchant's other apps: Omnisend allows 400 requests a minute per brand, for all endpoints and all integrations of that brand; Klaviyo gives an OAuth app its own quota per account, but private-key shops share one quota per account with every other private-key integration; Mailchimp allows 10 simultaneous requests per user; Intercom allows 10,000 calls a minute per app and 25,000 per workspace; Gorgias counts 80 requests per 20 seconds per account. Admission at the published limit would starve the merchant's other apps or be starved by them.
  5. Mailchimp limits concurrency, not rate. Admission for Mailchimp is a cap on simultaneous requests per shop, well under 10. A request that timed out on Joy's side still holds a slot at Mailchimp.
  6. Payload limits are per adapter. Klaviyo event: 5 MB, 100 KB a string, 400 properties. Klaviyo bulk import: 10,000 profiles, 5 MB, 100 KB a profile. Omnisend batch: 100 items. Drip batch: 1,000 records. Postscript: 50 custom properties a subscriber. Attentive: 200 keys an event type. The task itself stays small: Google's pages state the Cloud Tasks size limit as both 1 MiB and 100 KB, so the enqueue limit is set well under 100 KB and the payload is carried by reference.
  7. A 2xx can mean "accepted", not "delivered". Omnisend events and Postscript events answer 202 and are processed later; a Klaviyo bulk import and an Omnisend batch return a job id. The outcome log records "accepted" and "delivered" as different results.
  8. The worker never answers 429 or 503 to Cloud Tasks. Cloud Tasks reads those two codes as a signal to slow the whole queue, which would stall every shop and every app because one partner throttled one shop, and it would consume an attempt. On a partner 429 the worker creates a new task with a new attempt id and a scheduleTime after the adapter's wait, records the deferral, and answers 2xx. A deferral cannot be scheduled more than 30 days ahead. A new queue ramps up slowly, so the first cut-over must not start with a backlog.

Replay has an age limit per partner: Attentive does not start a Journey for an event older than 12 hours, and a LINE retry key is valid for 24 hours. "Retried until delivered" always means "inside that partner's limit" (decision 5 in section 8).

Producers keep using the existing topics for the fan-out decision. Only the delivery has to be durable, so the 212-case dispatcher never needs retry: true.

Health service. Each integration document gets a health map: connected, degraded, auth_broken, or disconnected, plus capability blocks for a missing scope. auth_broken is non-destructive: no revoke, tokens kept, delivery paused, merchant told once. Only a merchant disconnect revokes. Three classification rules come from the partners' documentation. First, a 403 is not a 401: Klaviyo permission_denied (a missing scope), a Mailchimp 403 for a user level that does not allow the endpoint, and Intercom api_plan_restricted are capability or plan blocks, never auth_broken and never a reason to refresh a token. Second, a 429 or a 5xx from a token endpoint is transient and never a dead grant; the refresh fence is therefore also a rate fence (Klaviyo allows 10 refreshes a minute). Third, a 410 from Omnisend means the API version was retired: that is "Joy must upgrade", not a merchant problem. The health state becomes the source of truth for an app only after every writer of that app (section 2.1) is migrated and fenced by installation generation; until then the app keeps its legacy truth and shows observational health. The shop flags stay as a derived cache, with a repair script for rollback.

Verifiers. One contract per inbound provider: verify the raw bytes, know where the secret came from, bind the request to an authenticated tenant, dedupe with the event lifecycle in mind, and accept the work durably before acknowledging. Each provider moves through a dual-accept window with per-shop counts of unsigned arrivals; the old registration is removed only after the new one is proven. The check is written per provider, because the schemes differ: Recharge is a plain SHA-256 of the API client secret joined to the body, not a keyed HMAC; Fera is a JWT in the URL; Loop is a base64 HMAC; Judge.me is a hex HMAC whose secret is Joy's one OAuth app secret, so the signature proves the sender and not the shop. Each enforcement window must be shorter than the time the partner waits before it gives up or removes the subscription: Shopify deletes an Admin-API subscription after 8 consecutive failures over 4 hours; Recharge deletes a webhook after 20 failures over 48 hours; Klaviyo disables a webhook that stays in error for more than 48 hours; Yotpo retries at least 10 times and only on a non-200; Loop makes six attempts and never retries a 401.

Credentials by reference. Messages and tasks carry an integration id, never a token. The worker loads the credential after checking shop, app and installation generation, so old queued work cannot run under a new installation. Secret Manager is used only for new app-level secrets. New partner keys are stored encrypted with a real key.

Partner API F1. Partners authenticate with a merchant-issued, per-partner, scoped key on REST v2. The list of permitted routes is default-deny. Quota is keyed by shop and partner with an aggregate shop cap; idempotency is namespaced by partner; a key can be revoked. Partner keys are a new principal type in one shared resolver, not a parallel middleware. OAuth comes later, only for a partner that requires it.

Success counters. Unsampled aggregate counters (source, accepted by Joy, accepted by the partner, delivered where the partner reports it, terminal, quarantined) per operation; diagnostics may be sampled. The legacy operation is measured before each cut-over. "Duplicate" means a duplicate external side effect, not a repeated attempt. Health and retention claims cover instrumented apps only.

5.3 The seven architecture decisions

  1. Integrations are declared in a code registry with per-app adapters. Rejected: definitions stored in Firestore. Every new app still needs code, so configuration as data saves nothing and gives no type safety and no test. Also rejected: keep the switches and add a checklist, because the seven unwritten flags show that checklists drift.
  2. Durable delivery runs on a dedicated Cloud Tasks queue, and the logical deliveryId is separate from the physical attempt task id. A task name only dedupes the enqueue (its reuse is blocked for up to 24 hours) and a task can run more than once. The deliveryId is a UUID. A partner 429 becomes a new task and a 2xx answer to Cloud Tasks, never a 429 answer (rule 8 in section 5.2). Rejected: retry: true on backgroundHandling*, because the dispatcher swallows errors and a retry would replay hundreds of unrelated, non-idempotent cases. Also rejected: "idempotency = task name".
  3. The integration document's health is the source of truth per app, and only after every writer of that app is migrated and fenced. Rejected: one global writer on day one, because seven writers stay live. Also rejected: deleting the shop flags (42 or more readers for Klaviyo alone) and live partner checks on read (the source of the self-inflicted 401 lines).
  4. Credentials are loaded by reference inside the worker; Secret Manager only for new app-level secrets. Rejected: moving every token to Secret Manager now. Thousands of per-shop tokens do not fit a per-secret model, and it would block the delivery work on an unrelated migration.
  5. Partners use merchant-issued scoped keys on REST v2 with a default-deny list of permitted routes; OAuth later. Rejected: reusing the MCP OAuth server now. It serves only public clients, and no shortlisted partner requires OAuth. Also rejected: reusing the shop secretKey or a constant partner key, which gives full access with no consent and no per-partner revocation.
  6. Migration is a strangler per app and operation. The routing flag is read at the producer. The consumer still checks generation and the breaker. Legacy handlers stay until their queue drains. A rollback routes new work only and drains or quarantines work that was already accepted. Rejected: one rollback flag for everything, and a rewrite of the fan-out in one merge request.
  7. auth_broken is non-destructive, and a missing scope is an operation-level block, not auth_broken. The partners' documentation supports this: Klaviyo answers a missing scope with HTTP 403 and code permission_denied, not with a 401. Rejected: reusing disconnectIntegration, which revokes the grant at the partner and nulls the tokens.

6. The work

Five groups, in order of execution. Every item states what to do, why, where in code, when it is done, and what it depends on. Section 7 says which wave each item is in.

Group 1 — Stop data loss now (before BFCM, 20/11 → 25/11)

Small, revertible changes and actions that need no deploy. They stop the leaks and protect merchants before the freeze.

1.1 Freeze the baseline and switch on three alerts IP-40 IP-47 P-01

1.2 Remove secrets and personal data from integration logs R-03 IP-15 IP-14

1.3 Stop the admin customer page from probing apps that are not connected R-01 P-03

1.4 Enforce review-webhook signatures where a secret exists, and count the rest IP-08

1.5 Classify dead Klaviyo and Gorgias credentials, log-only R-02 R-05

1.6 Check whether Omnisend contact sync downgrades consent R-18

1.7 Find the orders that lost points (ops, no tag)

1.8 Contact shops with dead credentials (support, no tag)

1.9 Take down the public Tapcart guide N-06a

1.10 Label integration tickets and freeze the ticket baseline IP-80

Group 2 — Fix existing integrations

One list per integration. Items that say "through the health service" need group 3 item 3.6.

Klaviyo

2.1 Mark a revoked Klaviyo grant as broken instead of retrying it R-02

2.2 Fix the two known Klaviyo retry bugs R-06 IP-23

2.3 Send the Klaviyo profile properties merchants segment on R-12

Omnisend

2.4 Keep events on after an Omnisend reconnect R-09

The Omnisend drops (172,229 contact updates, 10,404 events) are fixed by the delivery worker (3.4) and the bulk path (3.5), after the consent verdict (1.6).

Three facts about Omnisend's API change that work:

Mailchimp

2.5 Isolate shops in the Mailchimp cron and fail stuck jobs R-08 IP-24

Mailchimp's dead credentials (about 17,400 lines of 401 and 403 in 10 days) are handled through the health service (3.6), which states how Mailchimp's 401, 403 and 429 are read.

Check during the spike: the per-customer path adds every synced customer who is new to the list as subscribed, with no marketing-consent check (F/services/integrate/mailchimpService.js:81-83). This is the same class of problem as item 2.10. v8 adds no item for it; confirm with product whether it is intended.

Gorgias

2.6 Mark a failed Gorgias refresh as broken, and find out why it started R-05

Instagram

2.7 Show an expired Instagram session and ask for a reconnect R-14

2.8 Rotate the Instagram app secret IP-05

Drip

2.9 Stop Drip from looking disconnected after a transient error R-13

Attentive

2.10 Subscribe only customers who consented R-11 IP-29

Review apps

2.11 Backfill signature keys and tell Loox merchants IP-08

Chatty

2.12 Replace the Chatty shared constant key IP-03 IP-05

Shopify Flow

2.13 Make failing Flow actions fail visibly R-15

Outbound merchant webhooks

2.14 Count any 2xx answer as a delivered webhook R-16

2.15 Block webhook URLs and SMTP hosts that point inside the network IP-12

Retry and catch-up for failed webhooks is item 3.12.

Joy email

2.16 Try the custom domain only when one is verified R-17

2.17 Re-encrypt merchant SMTP passwords with a real key IP-06

Shopify order ingress and Admin API

2.18 Stop losing throttled Admin API calls R-19

2.19 Accept order webhooks durably, by reference IP-18b

2.20 Answer Shopify honestly when a webhook is not accepted IP-18

Joy Subscription

2.21 Fix the three Joy Subscription problems R-21

Note on the tag: v6 used R-20 for this item in one place and for the re-sync in another. From v7 on, R-21 is Joy Subscription and R-20 is only the re-sync (item 2.22).

Repair after the drops

2.22 Re-sync current state for shops that lost deliveries R-20

Group 3 — Build the shared integration infrastructure

Each step ships alone and is switched on per app and operation behind a gate. It is proven on one app first: Omnisend events for delivery, Klaviyo for health.

3.1 Outcome log for every integration call P-01 IP-41

3.2 Integration registry P-02

3.3 Read paths stop writing state P-03

3.4 Delivery worker P-04 IP-20 IP-22 IP-26

3.5 Bulk path for mass operations P-04b

3.6 Health service, per app P-05 IP-46

3.7 Replay and quarantine for operators P-06 IP-19

3.8 Inbound verifiers per provider P-07 IP-07 IP-08 IP-09

3.9 One auth contract for OAuth apps P-08 IP-10

3.10 Merchant health UI P-09 IP-51

3.11 Move the remaining profile syncs onto the worker P-10 R-10 IP-25

3.12 Retry and catch-up for outbound merchant webhooks P-11 IP-21

3.13 Contract tests and a staging sandbox P-13 IP-28

3.14 Instrument the remaining outbound and inbound paths P-14b IP-43 IP-44

3.15 The remaining alerts P-14c IP-47

3.16 Integration diagnostics for support P-14d IP-49

Deferred to next quarter: registry-driven dispatch (P-12); the BigQuery sink and hourly rollup (P-14a, IP-42), with the BigQuery part of the diagnostics page (IP-49); instrumentation of the remaining surfaces (IP-45); the fleet dashboard (IP-48) and the weekly report that needs it (IP-82); URL tokens for Yotpo and Review Rocket (Fera is verified through its JWT in 3.8).

Group 4 — New integrations

Why these apps and what competitors do is in research-new-integrations.md and research-potential-apps.md. In short:

App Competitors with a documented integration (of 9) What Joy copies or does better
Postscript 6: Smile, LoyaltyLion, Yotpo, Rivo, Gameball, Okendo Loyalty Push model like Rivo and Okendo: properties plus events, existing subscribers only, never subscribe anyone
Loop Returns 3: Smile, Yotpo, Rivo (all built by Loop, September 2026) Joy can lead: no competitor awards points when a customer picks an exchange over a refund
Tapcart 5: Smile, LoyaltyLion, Yotpo, Rivo, Okendo Loyalty Match the in-app block; fix the key exposure first
Okendo and other review apps 3 plus Okendo's own product The Flow template is the market-proven route (Rivo uses it)

4.1 Start the partner applications and measure demand (business development, no tag)

4.2 Flow review templates and the plan-gate fix N-01

4.3 Tapcart secure block N-06a

4.4 Loop listing N-04a

4.5 Subscription pack, sizing only N-08

4.6 Postscript beta N-03

4.7 Loop exchange bonus N-04b

Deferred: Zendesk sidebar app (N-05: demand unmeasured, Gorgias covers the Shopify-native helpdesk); Tapcart-native support (N-06b: only if Tapcart engages, needs F1); Okendo native connector (N-07: the Flow template is the end state unless Okendo asks); Postscript general availability; the subscription pack build.

Next wave, after 17/02: the subscription pack (Loop Subscriptions, Skio, Stay AI); a returns pack (Redo, AfterShip Returns) that reuses the exchange bonus; a Gladly helpdesk card, only after helpdesk demand is proven; zero-party Flow templates (Kno, Octane AI), each gated on a confirmed Flow trigger. None of these is outbound, so none waits for the delivery worker.

Not picked: Klaviyo SMS (covered by the Klaviyo sync); Yotpo SMS (shut down on 2025-12-31); native REVIEWS.io, Junip and Growave (covered by 4.2); Re:amaze and WhatsApp (on hold); Social Snowball, Aftersell, Smartrr native, ConnectPOS, Zalo ZNS, Vajro, Richpanel, wholesale apps, Rise.ai, Segment, Triple Whale, UpPromote and page builders.

Group 5 — Partner API and cleanup

The partner API is listed last because it is the outward-facing contract, but it is built early (Wave 1): the Loop listing and any Tapcart-native work wait for it.

5.1 F1, the partner-ready API N-02 IP-73 IP-76 IP-77

5.2 REST contract test in CI IP-72

5.3 REST v1: measure callers, publish the migration table IP-69 IP-70

5.4 REST v1 sunset: notice now, 410 later IP-71

5.5 Versioning policy and a channel to reach developers IP-74 IP-79

5.6 Smax: drain the stored batches or remove them IP-61

5.7 Label the link-only cards honestly IP-67

Deferred to next quarter: the npm publish pipeline (IP-75), the headless SDK contract test (IP-78), the REST v1 410.


7. Order and timing

7.1 Timeline

Wave Dates What goes in it
Wave 0, before the freeze Fri 20/11 → Wed 25/11 Group 1, all of it: 1.1 to 1.10. Deploys: 1.2, 1.3, 1.4, 1.5 (and the 1.6 fix only if a downgrade is confirmed)
Wave 0, BFCM freeze (dates proposed) Thu 26/11 → Tue 01/12 Branch work and read-only work, no deploy: the delivery-worker spike (3.4), the first registry entries (3.2), the Klaviyo writer inventory (3.6), the contract-test branch (5.2), the Tapcart session spike (4.3), F1 design, partner applications (4.1)
Wave 1 Wed 02/12 → Fri 18/12 Week 1: outcome log (3.1), small fixes (2.2, 2.4, 2.5, 2.9, 2.13, 2.14, 2.16), registry and read paths (3.2, 3.3), contract test merged (5.2), key backfill (2.11), Instagram secret (2.8). Week 2: delivery worker slice 1, Omnisend events (3.4). Week 3: Klaviyo health log-only (3.6, 2.1 first step) and order-webhook shadow mode (2.20), both observational. Across the wave: durable order acceptance (2.19), Joy Subscription parts a and b (2.21), Flow templates (4.2), F1 (5.1)
Wave 2 Mon 21/12 → Mon 18/01/2027 Holiday weeks (21/12, 28/12): branch work and log-only modes. From 04/01: Klaviyo health enforced (3.6, 2.1) with the badge and banner (3.10); order-webhook enforcement (2.20); delivery worker slice 2 (3.4), one adapter a week; bulk path (3.5); replay surface (3.7); verifiers (3.8); SSRF guard (2.15); Attentive consent (2.10); Tapcart secure block (4.3); Chatty key (2.12, if Chatty is ready); Klaviyo profile subset (2.3); health for Gorgias, Instagram and Mailchimp (3.6, 2.6, 2.7); Admin throttling first slice (2.18); Joy Subscription part c (2.21); per-shop re-sync (2.22); Loop listing (4.4); subscription sizing (4.5); Postscript spike and narrow slices (4.6)
Wave 3 Tue 19/01 → Wed 17/02/2027 (no work during the Tết break, 04/02 → 10/02) Remaining profile adapters (3.11); merchant-webhook retry (3.12); contract tests (3.13); instrumentation, alerts and diagnostics (3.14, 3.15, 3.16); SMTP re-encryption (2.17); auth contract (3.9); REST v1 work and developer channel (5.3, 5.4, 5.5); Smax and link-only cards (5.6, 5.7); Postscript build and beta (4.6); Loop exchange bonus (4.7). The third delivery adapter reaches the end of its soak here

Rules that hold in every wave

7.2 Safety gates

No production behaviour changes before its gate is met. A gate is a metric that exists before the deploy, a trigger that forces a rollback, and a rollback mechanism.

Gate Change it guards Metric that must exist first Rollback trigger Rollback mechanism
G-0 Wave 0 deploys: 1.2, 1.3, 1.4, 1.5 Alerts 1 and 2 of item 1.1 Any new error class Revert tag, under 30 minutes
G-1 Delivery worker slice 1, then each adapter (3.4, 3.11) Sent, retried, failed and ambiguous counts per app; queue depth; age of the oldest task; duplicate-deliveryId rate (Joy's attempts; a partner without working dedupe is covered by the quarantine count); the count of 429 or 503 answers the worker gave to Cloud Tasks, which must be zero Success over eligible logical deliveries below the legacy baseline, or any duplicate The producer flag routes new work back to the legacy path; accepted work drains on the new path or is quarantined, under 15 minutes; soak of at least 3 days before the next adapter
G-2 Health service, log-only to enforced (3.6) Would-be transitions per hour; false-positive rate against later successful calls; an alert when more than an agreed number of shops enter auth_broken within an hour False-positive rate above the threshold of decision 4 Flag off, then the flag-repair script (dry-run diff first)
G-3 Order webhooks, shadow to enforced (2.19, 2.20) Would-be non-2xx per topic for one week; answers slower than 5 seconds per topic; deliveries that share an event id with a different webhook id; non-2xx rate alert; subscription-existence check Non-2xx rate above the shadow baseline, or any missing orders/* subscription Revert tag; re-register subscriptions through the server path
G-4 Verifier enforcement per provider (3.8) Per-shop counts of unsigned arrivals; authenticated-installation coverage A legitimate delivery is rejected Per-provider dual-accept flag. The flag must be turned back inside the partner's own give-up window (section 5.2, Verifiers), or the partner removes the subscription
G-5 SMTP re-encryption (2.17); Tapcart key rotation (4.3) Decode failures; SMTP failures; list of affected shops Any spike in decode failures Read-both stays on; rotation is per shop, after the new block is live
G-6 2.13, 2.14, 2.16 and the 1.6 fix Flow run errors; send hook error rate; fallback lines per day; Omnisend contact and consent counts for a pilot shop An unexpected rise Revert tag

7.3 What must be true at the end of each wave


8. Decisions needed

Only the questions that block work.

# Decision Who decides By when What it blocks
1 Who staffs new-app work? Hire a developer, borrow one from another Joy app for 8 to 10 weeks, or run the reliability-first fallback described below Winn Before Wave 1 starts on 02/12 (derived from the wave plan; v6 set no date) 4.2, 4.4, 4.5, 4.6, 4.7, 5.1, and the Wave 2 slots of 2.18 and 2.21 part c, which v6 gave to this developer
2 F1 plan and pricing rule, as one pricing page: may partners call on Free; does partner traffic share the merchant's quota; do custom actions and headless stay Ultimate-only Winn and product 25/11 5.1 (F1), 4.4 and 4.7 (Loop), and Tapcart's reach (4.3, Tapcart-native)
3 Gorgias OAuth client: is Joy's client still valid, and what changed on 2026-11-05 01:46 UTC when invalid_client began on 8 accounts. The test that answers it: refresh one healthy account with the same client and compare. If the healthy account still refreshes, the cause is at account level and the 8 shops must reinstall the app Winn (owner of the Gorgias app credentials) 25/11, before support contacts the Gorgias shops 1.8 for Gorgias shops, 2.6
4 Health thresholds: the false-positive limit before enforcement; whether Joy ever auto-disconnects (none until decided); whether degraded is visible to merchants Winn, with product and support Before Klaviyo enforcement, the week of 04/01; the log-only week in Wave 1 supplies the numbers 3.6 enforcement (gate G-2), 2.1, 2.6, 2.7, 3.10
5 Ambiguous sends that trigger an SMS or an event: avoid a duplicate, avoid a loss, or hand to an operator to reconcile; and the maximum replay age, one value per partner (Attentive Journeys ignore events older than 12 hours; a LINE retry key lasts 24 hours). From v8 this also covers Omnisend events, which have no partner-side dedupe for automations, and PushOwl pushes Winn and product Quarantine is the default for Omnisend events from slice 1 in Wave 1 (week of 07/12). The decision itself is needed before the first adapter for a send that reaches a customer's phone, in Wave 2 4.6 (Postscript beta), the quarantine policy in 3.4, 3.7

If decision 1 is "nobody": the reliability-first fallback applies. Order-webhook work (2.19, 2.20), Joy Subscription (2.21), Admin throttling (2.18, ahead of 2.3) and the credential and tenant-boundary work keep their waves. F1 shrinks to a read-only pilot for one confirmed consumer. The Flow templates move to Wave 3 or next quarter. There is no Postscript, no Loop exchange bonus and no Klaviyo profile subset this quarter. The safety work (gates, durable acceptance, fences) is never cut.

Not decisions for now (user, 20/11): questions with no error evidence in the 18-day logs are not listed as blocking decisions: Smax; Omnisend API version; Attentive subscribe; Recharge verification secret. They are settled by the owner of the item when it is built (see each item's "Check during the spike" line).


9. Risks

  1. The breaker marks a healthy grant as broken. The likely causes are a reconnect that stored a new grant while old work was in flight, and a 429 from the token endpoint read as a dead grant (Klaviyo allows 10 refreshes a minute). A concurrent refresh by itself does not invalidate a Klaviyo refresh token. Mitigation: auth_broken is non-destructive, the refresh fence re-reads the stored refresh token before deciding and limits the refresh rate, one log-only week, a flag-repair script.
  2. Duplicates and ambiguity. A task name only dedupes the enqueue and Cloud Tasks can run a task twice. Mitigation: the logical deliveryId goes to the partner where the partner dedupes on it, which only Klaviyo and LINE document; for every other partner an ambiguous send is quarantined or is a state update that is safe to repeat. Exactly-once is not promised. A duplicate Omnisend event can fire a merchant's automation twice, so the quarantine volume for Omnisend must be watched from slice 1.
  3. Omnisend consent. If nonSubscribed downgrades contacts, every bulk path multiplies the harm. Item 1.6 gates all Omnisend batch and re-sync work.
  4. Tapcart key rotation breaks the merchant's old block and webview app together. It is done per shop, after the new block is live.
  5. Erasure against durable retries. A delayed delivery or a replay could re-export an erased customer. The worker checks an erasure fence before sending, and its delivery records must be covered by the existing customer-erasure path.
  6. Log cost. Success lines from the outcome log could be many times today's throttle lines, which already number more than 400,000. It needs a sampling rule, an ingestion budget and a dedicated bucket.
  7. Environment parity. New queues, collections, indexes and secrets missing on a staging slot make tests fail for the wrong reason.
  8. Visible failures raise tickets first. Items 2.13, the verifier enforcement, the banners and the deprecation headers show merchants errors that were silent before.
  9. Things outside Joy's control: nobody staffs new-app work; partner approvals (Postscript, Tapcart, Loop) or Chatty's client update slip; Omnisend retires /v5 with no published date (a retired version answers 410); a partner quota that Joy shares is used up by the merchant's other apps; Sendlane and PushOwl publish no limits.
  10. The delivery worker, the health service or the verifiers turn out bigger than planned after the spike. The response is to reduce scope, never the gates.

Appendix. Coverage of the existing plan's IP items

64 ids. One line per wave, then the items that are not scheduled, then the single out-of-scope line.