chore/integrations-roadmap, HEAD 2ff18f82dc.verify-partner-api-A.md for marketing and messaging partners, verify-partner-api-B.md for Shopify, helpdesk, social, reviews, subscriptions and the new apps). Statements the documentation contradicts are corrected, the documented numbers are added, and the partner constraints v7 did not state are written into section 5 and into the items they affect. v8 adds no new item; it adds three decisions (section 8, numbers 7 to 9). v7 is saved unchanged as roadmap-v7.md. The full list of changes is in apply-partner-verify-report.md.roadmap-v6.md.research-architecture.md, research-infra.md, research-existing-integrations.md, research-new-integrations.md, research-newapps-design.md, research-potential-apps.md, and the two partner-API verification reports named above. Where a research file and this roadmap disagree, this roadmap wins. Statements about a partner's API come from the partner's public documentation as read on 20/11; no partner API and no Shopify Admin API was called. Where the documentation is silent, the item says "Check during the spike".How to read citations and tags
F/ = packages/functions/src/, A/ = packages/assets/src/, ROOT/ = repo root. Every file:line was re-read at HEAD 2ff18f82dc for v7, and the citations added in v8 were re-read at the same HEAD.IP-xx = item of the existing plan. R-xx = fix to an existing integration. P-xx = shared infrastructure. N-xx = new integration. G-x = safety gate (section 7).The problem
ok reply as a failure (10,491 lines in an 11-day sample). Joy's own email tries a custom domain on every send for shops that have none (about 18,000 lines a day).authorization header; Recharge orders are logged with billing addresses; Apple Wallet request headers are logged (288 lines); every SMTP send logs the recipient and the subject.Behind all five is one cause: there is no integration module. Each app is hand-wired across about 30 files, on a message path that cannot retry, with no delivery id, no dead-letter queue, no health state and no success metric (sections 2 and 3). Compute and queue throughput are not the problem.
The work, in five groups
What changes after the rebuild (why rebuild instead of patching)
| Who | Today | After the rebuild |
|---|---|---|
| Merchant | Sees "Connected" while the credential has been dead for weeks; lost syncs are invisible (205k drops in 18 days, nobody told). | A badge and a Home banner say which app broke and what stopped, with Reconnect, Re-sync and one email (3.6, 3.10). |
| End customer | Order points can silently never arrive (344 orders); stale partner profiles show wrong balances in loyalty emails. | Orders are accepted durably before Shopify gets 200 (2.19, 2.20); every profile update is retried until it is delivered or quarantined, inside each partner's replay limit, and is never dropped silently (3.4). |
| Support | Searches Cloud Logging by hand, 18-day retention, no per-shop view. | Per-shop diagnostics with last success, 7-day outcomes, plain-language error and replay (3.7, 3.16); tickets labelled per integration (1.10). |
| Developer | A new app is ~32 files, 45 edit sites and 17 switch branches; no test catches a missed site; each app reinvents retry and rate limiting. | A new app is one registry entry plus one adapter; retry, rate limit, dedupe, health and metrics come from the platform; contract tests catch drift (3.2, 3.4, 3.13). |
| Business | SMS partners such as Postscript cannot be added safely (a duplicate is a second paid SMS); there is no partner API. | F1 lets Loop, Tapcart and Okendo build on Joy (5.1); Postscript ships on the delivery contract (4.6); Joy leads where no competitor is (4.7). |
The core change on one delivery. Today, when Omnisend answers 429 the handler sleeps, retries inside the same request, returns undefined, and the dispatcher returns 'ok': the update is gone after one log line (172,229 times in 18 days). After the rebuild, the producer writes a pending delivery with a stable deliveryId (a UUID) before acknowledging. The worker admits it within the shop's budget, which is set well below the partner's published limit because that limit is usually shared with the merchant's other apps. On a 429 the worker creates a new Cloud Task scheduled after the wait that this partner's adapter reads, without counting it as an attempt, and answers 2xx to Cloud Tasks. Omnisend sends no Retry-After header: the adapter reads retryAfter from the 429 body when it is there and otherwise backs off with jitter. The deliveryId goes into the partner's dedupe field only where the partner documents one that works (Klaviyo unique_id, LINE X-Line-Retry-Key). Omnisend has none for events that trigger automations, so an Omnisend event whose send timed out is quarantined for operator replay instead of being resent blindly. A 401 follows the same idea: auth_broken stops the calls, keeps the tokens and tells the merchant. A 403 for a missing scope or plan is not a 401: it blocks one capability and leaves the connection alone.
One row per integration that has runtime code, ordered by merchants connected. Merchants connected = distinct shops with an active connection whose Joy install is still live (shops.isInstalled == true, out of 20,835 installed shops); incl. uninstalled adds shops that have since removed Joy. Counted read-only from prod Firestore on 20/11/2026, aggregates only: integration-usage-counts.md. Flow, Joy Subscription and Wallet come from shop-level flags; the rest from the integrations collection (isConnected == true). "not tracked" = no connection record exists to count (Joy-provided APIs, Smax, Chatty, Joy Wishlist, Klaviyo Reviews, Facebook Page, order webhooks). Installed is an upper bound: frozen or closed stores keep isInstalled: true. The Integrations page also shows 17 link-only cards (for example Okendo, REVIEWS.io, Skio); they have no runtime connection and are not listed here.
| App | Category | Merchants connected | incl. uninstalled | How it connects (direction, transport) |
|---|---|---|---|---|
| Shopify Flow | Shopify platform | 1,456 | 2,097 | Outbound: Flow triggers through the Admin API. Inbound: HMAC-signed Flow action calls over HTTP |
| Judge.me | Reviews | 1,114 | 1,944 | Inbound: unsigned review webhooks over HTTP. Outbound pull: a daily review backfill (OAuth) |
| Klaviyo | Email/SMS marketing | 790 | 1,029 | Outbound: loyalty events, profile changes and bulk re-sync, Pub/Sub then Klaviyo REST API (OAuth or a private key). Inbound: see Klaviyo Reviews |
| Joy Subscription | Subscriptions (Avada app) | 428 | 631 | Inbound: signed webhook over HTTP. Outbound: plan and install calls over HTTP |
| Air Reviews | Reviews | 332 | 517 | Inbound: review webhook over HTTP with an HMAC header |
| Social | 239 | 282 | Inbound: signed Meta webhook over HTTP, then Pub/Sub. Outbound: Graph API calls (OAuth) | |
| Google Wallet | Wallet | 178 | 214 | Outbound: pass object updates, Pub/Sub then Google Wallet API. Inbound: save callback over HTTP |
| PushOwl | Web push | 154 | 198 | Outbound: push notifications for loyalty events, Pub/Sub then PushOwl API (partner key and a per-shop token) |
| Omnisend | Email/SMS marketing | 149 | 202 | Outbound: events, contact updates and bulk batches, Pub/Sub then Omnisend REST API (OAuth) |
| Apple Wallet | Wallet | 139 | 169 | Outbound: APNs push when a customer changes. Inbound: PassKit web service and pass downloads over HTTP |
| Yotpo | Reviews | 97 | 156 | Inbound: unsigned review webhook over HTTP |
| Mailchimp | Email marketing | 75 | 96 | Outbound: audience member updates, Pub/Sub then Mailchimp REST API (OAuth); bulk batches polled by a 5-minute cron |
| Gorgias | Helpdesk | 58 | 61 | Outbound: customer data for the helpdesk widget, Pub/Sub then Gorgias REST API (OAuth) |
| Fera | Reviews | 50 | 88 | Inbound: unsigned review webhook over HTTP |
| Recharge | Subscriptions | 23 | 28 | Inbound: unauthenticated order webhook over HTTP. Outbound: discount sync through a Cloud Task (API token) |
| Sendlane | Email marketing | 13 | 13 | Outbound: bulk list sync only, on connect or manual re-sync (API token) |
| Loox | Reviews | 10 | 10 | Inbound: review webhook over HTTP, HMAC-signed when the shop has a key |
| LINE | Messaging | 5 | 5 | Outbound: push messages, Pub/Sub then LINE Messaging API. Inbound: signed Messaging API webhook |
| Review Rocket | Reviews | 4 | 4 | Inbound: unsigned review webhook over HTTP |
| Drip | Email marketing | 3 | 3 | Outbound: subscriber fields, Pub/Sub then Drip REST API (OAuth); bulk continued by an hourly cron |
| Intercom | Helpdesk | 2 | 2 | Outbound: contact attributes, Pub/Sub then Intercom REST API (OAuth or a personal access token) |
| Attentive | SMS marketing | 0 | 1 | Outbound: events, custom attributes and bulk sync, Pub/Sub then Attentive REST API (OAuth) |
| Stamped | Reviews | 0 (no connected doc) | 0 | Connect-time credential check only; no review ingestion path exists |
| Smax | Messaging | not tracked | — | Outbound: point and tier changes, Cloud Task then Smax API (one global token for all shops) |
| Chatty | Helpdesk / chat (Avada app) | not tracked | — | Both: Chatty calls /integrate/oauth/connect with one shared constant key; Joy calls Chatty with the same key |
| Klaviyo Reviews | Reviews | not tracked | — | Inbound: review webhook over HTTP, HMAC-signed when a secret is stored |
| Joy Wishlist | Wishlist (Avada app) | not tracked | — | Inbound: events over HTTP (JWT). Outbound: wishlist API calls; bulk chunks through a Cloud Task |
| Facebook Page | Social | not tracked | — | Inbound: signed Meta webhook over HTTP, then Pub/Sub. Outbound: Graph API calls (OAuth) |
| Shopify order webhooks | Shopify platform | not tracked | — | Inbound: HMAC-signed webhooks over HTTP, then one Cloud Task per order |
| Outbound merchant webhooks | Joy-provided API | not tracked | — | Outbound: HMAC-signed HTTP calls to the merchant's URL, Pub/Sub for activity topics, Cloud Tasks for program topics |
| Email sending (Mailgun custom domain, merchant SMTP, default SMTP) | Email delivery | not tracked | — | Outbound: every notification, Pub/Sub then SMTP or Mailgun |
| REST API v2 (and v1) | Joy-provided API | not tracked | — | Inbound: merchants and partners call Joy over HTTPS with an app key and a secret key |
| Headless API | Joy-provided API | not tracked | — | Inbound: headless storefronts and mobile apps call Joy over HTTPS with a headless token |
| MCP | Joy-provided API | not tracked | — | Inbound: AI clients call Joy through the MCP server (OAuth 2.1), which calls REST |
Two findings from the count: (1) the two connection sources disagree (Klaviyo 790 by integration doc vs 761 by shop flag; Judge.me 1,114 vs 1,216), which is the "four answers to connected?" problem of §2.1, now measured; (2) many connections sit on shops that uninstalled Joy (Judge.me 830, Klaviyo 239), consistent with calls still made on dead credentials (§4.1).
Source: research-infra.md section 8 (every row there is listed here), plus Chatty from its section 1.5.
There is no integration module. One app is spread over about 30 files. Attentive, the closest template for a new outbound app, touches about 32 files and 45 edit sites, 17 of them branches in switch or if chains (a count of today's touch points, not a commit diff).
Constants. The app id lives with the loyalty-program constants, for example ATTENTIVE_APP at F/const/programs/integrationPrograms.js:11. The event list and the message and job types are separate constants (F/const/integrations.js:338, F/const/integrations.js:376). Id values use four casings and one app has two ids, so nothing can be derived from the id.
Switch chains. Adding an app means adding a branch in each of these places:
F/repositories/integrationsRepository.js:623-697, F/repositories/integrationsRepository.js:699-790), plus the side effects after a save (F/repositories/integrationsRepository.js:806-897);F/controllers/integrationController.js:317, F/controllers/integrationController.js:479-541, F/controllers/integrationController.js:618, F/controllers/integrationController.js:673);F/services/triggerIntegrationAppService.js:114-208, F/services/triggerIntegrationAppService.js:354-366, F/services/triggerIntegrationAppService.js:453-533, F/helpers/integrations.js:8-17);switch (type) with 212 cases in a 9,933-line file (F/handlers/pubsub/subscribeBackgroundHandling.js:745);F/handlers/schedule/enqueueHandler.js:165-212);A/config/integration/appList.js:187, A/const/integrationStatusFields.js:27-47).No test fails when one of these sites is missed.
Connection flags. "Is this app connected?" has four answers that can disagree:
statusConnected* (19 names), used as a cheap gate by the fan-out (F/services/triggerIntegrationAppService.js:453-476);isConnected, shown by the admin list (A/pages/Integration/Integration.js:62-79);statusConnectedDripApp and statusConnectedJudgeMeApp (A/const/integrationStatusFields.js:32-42); the backend writes statusConnectedDrip and statusConnectedJudgeApp;F/services/ai/tools/integrationToolService.js:183-262).The seven writers of connection state. At least seven code paths change whether an app counts as connected, and they do not agree on which fields to write:
F/controllers/integrate/klaviyoController.js:23, F/controllers/integrationController.js:688);F/controllers/integrationController.js:562-601, F/repositories/integrationsRepository.js:699-790);F/repositories/integrationsRepository.js:806-897);F/services/klaviyoService.js:349-354 calls F/repositories/integrationsRepository.js:899-933; the revoke is at :910-925);isConnected: false and an error (F/services/intercomService.js:115-121);F/services/dripService.js:371-378);F/controllers/integrationController.js:457-550).A read path also changes state: the admin's live Klaviyo check flips isConnected only in the HTTP response (F/controllers/integrationController.js:322-341).
sendEventToIntegrations for an event (about 40 call sites) or updateCustomerToIntegrations for a profile change (about 14 call sites).F/services/triggerIntegrationAppService.js:83-98). If the publish fails, the error is logged and the event is lost at the source (F/services/triggerIntegrationAppService.js:99-103).backgroundHandlingLight (events) or backgroundHandlingSmall (profiles). Both subscriptions run with retry: false (F/entries/pubsub-background.js:39-47).'ok' whatever happened (F/handlers/pubsub/subscribeBackgroundHandling.js:1717-1727).Promise.all and swallows per-app errors (F/services/triggerIntegrationAppService.js:209-213, F/services/triggerIntegrationAppService.js:367-371).F/helpers/klaviyoTaskQueue.js:27-50, F/helpers/intercomTaskQueue.js:25-36, F/helpers/lineTaskQueue.js:21-40).Profile changes have two more drop points: a change is skipped while a bulk tag, metafield or tier-launch run is active (F/services/triggerIntegrationAppService.js:326-336), and each per-app publisher skips while the integration's bulk sync is running (F/repositories/customerRepository.js:7236, F/repositories/customerRepository.js:7265, F/repositories/customerRepository.js:7301).
Partner and Shopify webhooks arrive on HTTP functions and are processed inline in the request. Each source has its own verifier, written differently:
F/middleware/webhook/verifyHook.js:40-52); a bad HMAC is answered with HTTP 200 (F/middleware/webhook/verifyHook.js:135-140); the dedupe log is written before the work is queued (F/middleware/webhook/verifyHook.js:141-150). The dedupe prefers X-Shopify-Event-Id and is scoped by shop, not by topic (F/repositories/webhookLogRepository.js:20-37). Shopify documents X-Shopify-Webhook-Id as the key for one delivery and X-Shopify-Event-Id as shared by every delivery that came from one merchant action.F/middleware/webhook/verifyHookReviewApp.js:17-57), and a processing error is answered with HTTP 200, so the partner never retries. Two of the four do sign: Judge.me sends a JUDGEME-HMAC-SHA256 header and Fera sends a JWT in the jwt URL parameter. Yotpo documents no signature, and Review Rocket has no public documentation.F/middleware/webhook/verifyHookLoox.js:70-84, F/middleware/webhook/verifyKlaviyoWebhook.js:108-109).F/routes/apiHookV1.js:627).F/const/integration/chatty.js:1).F/helpers/joySubWebhookVerification.js:23), Instagram and Facebook (F/helpers/social/verifyWebhookSignature.js:12-25).Array.includes over two accepted forms, and a bad HMAC is answered with HTTP 200 (F/middleware/webhook/verifyHookShopifyFlow.js:26-43). The action_run_id marker is stored before the handler runs (F/middleware/webhook/verifyHookShopifyFlow.js:44-62).Order webhooks are the one inbound flow that is queued: the middleware puts the whole request state (shop document plus order payload) into a Cloud Task (F/middleware/webhook/enqueueWebhookToCloudTasks.js:37-41). If that enqueue fails, the error is logged and Shopify still gets a success answer (F/middleware/webhook/enqueueWebhookToCloudTasks.js:46-58).
integrations collection (F/repositories/integrationsRepository.js:63). The repository has no encrypt or decrypt call.F/services/triggerIntegrationAppService.js:83-98, F/helpers/klaviyoTaskQueue.js:44-45). The shop's REST secretKey is copied into the outbound-webhook message (F/handlers/onCreateActivity.js:84).flowchart LR
subgraph ADMIN["Admin UI"]
CARDS["Integrations list and detail pages"]
CUST["Customer page sync status"]
end
subgraph CODE["Per-app wiring, no module"]
IC["integrationController<br/>4 app switches"]
IR["integrationsRepository<br/>connect, disconnect, side effects"]
TIS["triggerIntegrationAppService<br/>fan-out lists and gates"]
end
subgraph STATE["State in Firestore"]
DOC[("integrations doc<br/>isConnected plus tokens in clear")]
FLAG[("shop flags<br/>statusConnected x19")]
RL[("integrationRateLimits<br/>3 apps only")]
end
subgraph ASYNC["Async path"]
PS["Pub/Sub backgroundHandling<br/>retry off"]
DISP["Dispatcher, 212 cases<br/>always returns ok"]
CT["Cloud Tasks enqueueSubscriberv2<br/>one attempt"]
end
BIZ["Loyalty events and customer changes"] --> TIS
CARDS --> IC
CUST -->|"live partner call per view"| IC
IC --> IR
IR --> DOC
IR --> FLAG
TIS -->|"message carries tokens"| PS
PS --> DISP
DISP --> SVC["Per-app services<br/>own retry and rate limit"]
SVC --> RL
SVC -->|"hand-written re-enqueue"| CT
CT --> SVC
SVC --> PARTNERS[("Partner APIs")]
SVC -->|"error: log one line, drop"| LOST["Lost work"]
HOOKS["Partner and Shopify webhooks"] --> VER["One verifier per source<br/>4 unsigned, 2 fail-open"]
VER --> INLINE["Inline handlers<br/>HTTP 200 on failure"]
VER -->|"orders only"| CT
Everything below is defined in F/entries/*.js and was re-read at HEAD.
HTTP functions. The integration endpoints set no region, so they run in us-central1.
apiHookV2: inbound partner and Shopify webhooks, 30-second timeout, 1 GiB (F/entries/http-webhooks.js:33-34). Shopify allows 5 seconds for the whole request, Recharge and Klaviyo Reviews 5 seconds, Loop 10 seconds.apiHookV1Passv2: Apple and Google Wallet (F/entries/http-webhooks.js:27-28).apiShopifyFlowv2: Flow actions, 10-second timeout (F/entries/http-integrations.js:24-27), the same as the 10 seconds Flow waits before it closes the connection and resends.apiIntegratev2: /integrate/** for Joy Subscription, Joy Wishlist and Chatty (F/entries/http-integrations.js:18-19).restApiv2: REST v1 and v2 (F/entries/http-rest.js:25-26).Pub/Sub topics and retry settings.
backgroundHandlingLight, backgroundHandlingSmall, backgroundHandlingMedium: retry: false (F/entries/pubsub-background.js:39-52). All integration events and profile syncs travel here.backgroundHandlingOrdered: retry: true with a dead-letter handler (F/entries/pubsub-background.js:68, F/entries/pubsub-background.js:74-77). The shared dispatcher still catches every error and returns 'ok', so a failed case is not redelivered.recursive (bulk re-sync), webhookSend (outbound merchant webhooks) and sentEmailNotification (email): no retry option set (F/entries/pubsub-background.js:33-36, F/entries/pubsub.js:36-37, F/entries/pubsub.js:67-70).reliableJobs and birthdayReward retry in a way that reaches the handler (F/entries/pubsub-background.js:80-90). No integration uses them.Cloud Tasks. One shared queue, enqueueSubscriberv2: 120-second timeout, 1 GiB, retryConfig.maxAttempts: 1, no rateLimits (F/entries/tasks.js:16-23). It carries order webhooks, the Klaviyo, Intercom and LINE re-enqueues, bulk retries, program webhooks and the Recharge discount sync. Because the queue allows one attempt, every retry is a hand-written re-enqueue. A dedicated, rate-limited queue with real retry settings exists for another feature and is the template for integrations (F/const/pointExpiration.js:39-43).
Schedulers. Mailchimp job polling and the Klaviyo and Omnisend import-job checks run inside the 5-minute cron (F/handlers/schedule/frequentOperations.js:14-18); Drip sync runs inside the hourly cron (F/handlers/schedule/hourlyOperations.js:16); the Judge.me backfill runs daily at 01:00 (F/entries/schedule.js:43-44). No cron refreshes OAuth tokens. No cron re-sends failed merchant webhooks: the catch-up job publicSendWebhookForActivitiesMissing.js exists but is not registered. No cron drains Smax batches.
Firestore collections.
integrations: one document per shop and app, holding settings, isConnected, sync status and the tokens (F/repositories/integrationsRepository.js:63).integrationRateLimits: throttledUntil per shop and app with a 1-hour TTL and a 30-second per-instance cache; used by Klaviyo, Attentive and Intercom only (F/repositories/integrationRateLimitRepository.js:10-60).jobs: bulk-sync jobs, shared with CSV import history (F/repositories/jobRepository.js:17).webhooks and webhookInternalLogs: outbound merchant webhooks. webhookLogs: Shopify webhook dedupe, kept 30 days (F/helpers/utils/getTimeToLiveExpire.js).Deploy. In CI only staging 12 and production deploy Firestore indexes, so a new queue, collection, index or secret must be provisioned on every staging slot that is used for a test.
Email and SMS marketing, outbound (Klaviyo, Omnisend, Mailchimp, Attentive, Drip, Sendlane, PushOwl)
retry: false Pub/Sub topics, then the per-app service. Klaviyo profiles re-enqueue on the shared Cloud Tasks queue; Klaviyo mass operations use bulk collectors; Omnisend bulk uses batches.F/services/klaviyoService.js:349-354). A failed Klaviyo OAuth refresh does not: the invalid_grant branch is unreachable because the HTTP client throws on a 4xx first (F/services/klaviyoService.js:717-758). Klaviyo treats 401 and 403 alike (F/services/klaviyoService.js:328), so on an OAuth shop a 403 for a missing scope starts a token refresh that cannot help, with no lock and no backoff (F/services/klaviyoService.js:333-345). Omnisend, Attentive and Mailchimp throw, log and keep calling. Drip disconnects on any error, on the bulk path and on the per-customer path (F/services/dripService.js:244-245).F/handlers/schedule/enqueueHandler.js:544-549). A Klaviyo event is dropped (F/services/integrateKlaviyoService.js:779-781). An Omnisend contact update sleeps inside the handler up to 3 times, then is dropped (F/services/integrate/syncOmnisendService.js:373-399). An Omnisend event is dropped (F/services/integrate/omnisendService.js:66-80). An Attentive profile returns "queued" but queues nothing (F/repositories/integrationsRepository.js:580-585).Retry-After header. Klaviyo's limits are per endpoint tier, but Joy keeps one throttle flag per shop for all endpoints (F/services/klaviyoService.js:366-380). Omnisend's handler reads x-rate-limit-reset (F/services/integrate/omnisendService.js:61-70), a header the current Omnisend API no longer documents. Attentive reads a retry-after header that Attentive does not document, so it always uses its fixed default (F/services/attentiveService.js:77-85).F/services/pushOwlService.js:45-46).Helpdesk and messaging (Gorgias, Intercom, LINE, Smax, Chatty)
undefined and the caller then throws a TypeError on every call (F/services/gorgiasService.js:198-238); its rate limiter lives in the memory of one instance (F/services/gorgiasService.js:27-31).F/services/intercomService.js:115-121); 429 and 5xx are retried up to 5 times.Retry-After header (F/services/line/lineApiService.js:157-177). No X-Line-Retry-Key is sent, so a Cloud Task that runs twice sends the message twice. v7 said a retried 429 could duplicate the message; that was wrong, because a 429 means LINE did not accept it.Reviews, inbound (Judge.me, Yotpo, Loox, Stamped, Fera, Review Rocket, Air Reviews, Klaviyo Reviews)
apiHookV2, inline in the request, with a generic dedupe by shop, app and review id.Subscriptions and wishlist (Recharge, Joy Subscription, Joy Wishlist)
F/controllers/integrationController.js:769-778). Recharge waits 5 seconds, resends a failed delivery 20 times over 48 hours and then deletes the webhook, so the status this handler really returns must be checked on staging (item 3.8).F/middleware/webhook/verifyJoySubWebhook.js:63-91).Social (Instagram, Facebook Page)
backgroundHandlingLight. The token is refreshed on demand before a call, and only when less than 24 hours of its 60-day life remain (F/services/instagram/instagramApiService.js:63-77). A shop that does not use the integration inside that last day passes the expiry, and Meta then refuses the refresh for good. When the refresh fails the old token is returned and the call fails, on every use (F/services/instagram/instagramApiService.js:41-47, F/services/instagram/instagramApiService.js:103-105).Wallet (Apple Wallet, Google Wallet)
authorization: ApplePass are still logged (F/controllers/loyaltyPassController.js:302-306, F/controllers/loyaltyPassController.js:459-463).Email sending (Mailgun custom domain, merchant SMTP, default SMTP)
sentEmailNotification topic; a failed send is not retried.F/services/email/sendEmailService.js:429-462). It is the template for the health service.F/services/email/sendEmailService.js:129-131).Shopify (order ingress, Admin API, Flow)
F/services/shopify/shopifyMetaFieldService.js:3453, F/services/shopify/shopifyMetaFieldService.js:3515-3519); a throttled tag write is logged and dropped (F/services/shopifyService.js:3302-3310). Shopify reports a GraphQL throttle as HTTP 200 with errors[].extensions.code equal to THROTTLED, never as a 429, so code that classifies by HTTP status does not see it.success: false, so Flow records a success (F/controllers/shopifyFlowController.js:1198-1203). Flow resends an action after a 5xx, a 429, a 202 or a 10-second wait, for up to 36 hours; because the action_run_id is already stored, Joy answers each resend "duplicated" with HTTP 200.Joy-provided APIs (REST v2, headless, MCP, outbound merchant webhooks)
F/handlers/onCreateActivity.js:81-84); a 2xx answer with a non-JSON body is recorded as a failure (F/services/webhookService.js:89-95); a merchant 5xx is never retried. Program webhooks already retry 6 times through Cloud Tasks (F/services/program/programWebhookDeliveryService.js:12-15).For scale, yes. For reliability, no. Cloud Functions, Pub/Sub and Cloud Tasks handle far more traffic than 5 to 8 more partners add. The Cloud Tasks limits (500 dispatches per second per queue, 1,000 queues per region by default) are not a problem, but they rule out one queue per shop. What is missing is everything that makes a delivery safe:
| Capability | Verdict | Evidence |
|---|---|---|
| Durable queue with retry and backoff | Exists, but integrations do not use it | Integration topics are retry: false; the shared queue allows one attempt; errors are swallowed (F/services/triggerIntegrationAppService.js:209-213) |
| Rate limiting that honours each partner's own retry signal | Partial | Three apps share integrationRateLimits; Omnisend sleeps in the handler; Gorgias limits per instance; no queue-level rateLimits. Only Klaviyo documents a Retry-After header; Omnisend, Attentive, LINE, Postscript, Drip and Mailchimp do not |
| Idempotency key per delivery | Missing | Klaviyo's unique_id falls back to a random id (F/services/integrateKlaviyoService.js:453); no integration task has a name |
| Dead-letter queue and replay | Missing for integrations | Only the ordered topic has one, attached by a manual script that is not in CI (ROOT/scripts/setup-pubsub-ordering.sh:136) |
| Auth circuit breaker that tells the merchant | Missing except for SMTP | Section 3.2; 61,240 Klaviyo refresh-failure lines in 18 days |
| Success and failure metrics per app | Missing | Apps log failures as free text and do not log successes |
| Inbound signature verification | Partial | Four sources with no check in Joy (two of them, Judge.me and Fera, do sign), two fail-open, one unauthenticated (section 2.3) |
| Secrets handling | Partial | Tokens in message payloads and in logs; one constant key in source |
The prod logs in section 4 show the cost of each gap.
Method. Cloud Logging entries.list on resource.type="cloud_run_revision" (all regions), 2026-11-02 → 2026-11-19, paginated to exhaustion, read-only with the support service account. The _Default bucket keeps 18 days, so the counts below are the frozen baseline: the raw entries expire, these counts do not.
Each row gives what the logs show, the cause type as "trigger → why it persists", the code evidence, and the item in section 6 that fixes it.
Klaviyo
| Finding | What the logs show | Trigger → why it persists | Evidence in code | Fixed by |
|---|---|---|---|---|
| Auth probe | Unauthorized Klaviyo access 25,438 lines (exact) = 12,719 failed calls, each logged again by its caller; 810 shop ids; 24,246 lines on apiv2/apisav2, 1,192 on enqueuesubscriberv2 (the sync path, not the probe) |
CODE (admin page probes unconnected shops) → CODE (a read path calls disconnectIntegration) |
A/pages/Customer/Details/IntegrationStatus.js:46-54; F/helpers/integration/klaviyo/checkKlaviyoSyncStatus.js:15-27; F/services/klaviyoService.js:349-354 |
R-01 |
| Dead OAuth | Failed to refresh token. 61,240 lines (exact) = 27,166 failed refreshes, the rest are callers logging the same failure; 44 shops; top 3 ≈ 40k lines |
PARTNER/CONFIG (merchant revoked or uninstalled; some may have uninstalled Joy itself) → CODE (invalid_grant branch unreachable, axios throws on 4xx; the branch would also revoke and null tokens) + INFRA (no breaker) |
F/services/klaviyoService.js:717-758 (revoke at :732-748) |
R-02, P-05 |
| 429 burst | 429 6,241 (exact; 5,466 on 11-13). Throttle/re-enqueue lines ≥ 400,000 (capped), 187k from one shop in 3 hours. 22,479 profile syncs dropped after 3 tries (exact; 7 shops, 21,967 from one shop on 11-12/13). 498 bulk batches dropped after their one retry was throttled again, 46 more on a refresh failure. | INFRA (mass operation for one shop) → INFRA (the throttle gate re-enqueues and each deferral counts toward MAX_KLAVIYO_RETRIES = 3; the throttle cache is per instance for 30 s). Production-version-dependent: demotion has used bulk collectors with skipKlaviyo since a37aecb835 (07/10), so the drops are on the per-customer klaviyoSync path the collectors do not cover |
Collectors F/handlers/pubsub/subscribeBackgroundHandling.js:2563-2564, 2638; drop F/handlers/schedule/enqueueHandler.js:104, 544-549; cache F/repositories/integrationRateLimitRepository.js:30-59; bulk drop F/services/integrateKlaviyoService.js:991-1030 |
P-04b, P-04 |
| Missing scope | MISSING_EVENTS_WRITE_SCOPE 1,326 lines (exact) = 663 failed events (two lines each), 3 shops |
CONFIG (key lacks events:write) → UX (merchant not told). Event capability only: profile sync still works |
F/services/klaviyoService.js:349-350 |
P-05 (capability block), P-09 |
Omnisend
| Finding | What the logs show | Trigger → why it persists | Evidence in code | Fixed by |
|---|---|---|---|---|
| Contact 429 | error updateOmnisendContact 177,491 (exact): 172,229 rate-limited and dropped, 5,262 other; 63 shops (top 3 = 157k); spikes 11-08 (52k), 11-13 (23k), 11-16 (84k) |
INFRA (mass updates fan out per contact) → INFRA (retry sleeps inside the handler, then returns undefined) |
F/services/integrate/syncOmnisendService.js:373-399 |
P-04, P-04b (after R-18) |
| Event 429 | 10,404 events dropped (exact), 4 shops | PARTNER limit → INFRA (sendEvent throws on 429, caller logs and returns; no dedupe id sent, and Omnisend's eventID does not dedupe events that trigger automations) |
F/services/integrate/omnisendService.js:66-80, 163-188; syncOmnisendService.js:509 |
P-04 slice 1 |
| Probe | shop not connect omnisend 15,276 (exact) |
CODE (same admin probe; cheap, no network) | F/services/integrate/syncOmnisendService.js:37 |
R-01 |
| Consent (code only, no log signal possible) | Every contact upsert sends channels: {email: {status: 'nonSubscribed'}}, or sms for phone-only contacts. Whether this downgrades subscribed contacts is unproven. If it does, each sync shrinks merchants' audiences |
SEC/CODE | F/services/integrate/omnisendService.js:221-231 |
R-18 |
Mailchimp, Gorgias, Instagram (dead credentials)
| App | What the logs show | Trigger → why it persists | Evidence in code | Fixed by |
|---|---|---|---|---|
| Mailchimp | Sample 2026-11-10 → 19 (per-day cap): 401 ≈ 12.2k, 403 ≈ 5.3k, 429 ≈ 2.2k across Error checking email in list (18,493 lines) and Error updating contact in Mailchimp (2,889 lines); all 21,382 lines carry the merchant's authorization header |
PARTNER/CONFIG (revoked key) → INFRA (no breaker); SEC (whole error object logged) | F/services/integrate/mailchimpService.js:196 |
R-03, P-05 |
| Gorgias | Sample (cap reached at 42 hours, 18/11 → 19/11): refresh fails with invalid_client on 8 accounts, ≈ 3,000 failed refreshes/day; each failure then throws Cannot destructure property 'accessToken' (≈ 5,900 lines/day; ≈ 8,900 error lines/day in total). The first failure in the retained logs is 2026-11-05 01:46 UTC; there are none on 02/11-04/11 |
PARTNER/unknown (something changed on 05/11) → CODE (handleRefreshToken returns undefined) + INFRA (no disconnect). Unproven whether all Gorgias shops fail |
F/services/gorgiasService.js:198-238 |
R-05, P-05 |
| Expired or invalidated session: ≈ 6,300 error lines in a 14-day capped sample (05/11 → 19/11), 140-730 lines/day on every day; 1,492 failed token refreshes | CONFIG (session expired, or merchant changed password) → INFRA + UX (the refresh is retried on every use of the integration, there is no cron and no reconnect prompt) | F/services/instagram/instagramApiService.js:41-47, 103-105 |
R-14 |
Reviews and subscriptions
| App | What the logs show | Trigger → why it persists | Evidence in code | Fixed by |
|---|---|---|---|---|
| Loox | 17 Invalid webhook signature in 18 days |
Verification works where a key exists (401 returned). Legacy installs without a key skip verification | F/middleware/webhook/verifyHookLoox.js:70-84; key created on connected save F/repositories/integrationsRepository.js:80-82 |
IP-08, P-07 |
| Judge.me | ~2.5k review hooks (2,475), no failure clusters | Webhook unsigned (IP-07) | F/middleware/webhook/verifyHookReviewApp.js:17-57 |
P-07 |
| Recharge | Order logged as JSON with billing address | SEC; endpoint unauthenticated (IP-09) | F/services/integrate/rechargeService.js:281, 296 (route F/routes/apiHookV1.js:627) |
R-03, P-07 |
| Joy Subscription | 724 Ignoring unhandled topic: subscription_billing_attempts/failure (18 days); 216 HTTP 401 in Joy Sub API responses |
PRODUCT (topic subscribed but not handled) / CONFIG (401) | F/controllers/joySubWebhookController.js:84 |
R-21 |
Joy's own outbound channels
| Channel | What the logs show | Trigger → why it persists | Evidence in code | Fixed by |
|---|---|---|---|---|
| Outbound merchant webhooks | 20,057 send hook error lines in ~11 days (capped sample, 09/11 → 19/11) on webhooksubscriberv2. 10,491 are endpoints that answered the plain body ok; ~920 answered other plain text (skipped, updated, created, …); ~8,600 are error pages or network errors (Internal Server Error 6,678, Service Unavailable, upstream, fetch failed). The HTTP status is not logged, so "2xx" is inferred from the body |
(a) CODE: a successful 2xx with a non-JSON body becomes status 500 → R-16. (b) PARTNER trigger (merchant endpoint down) → INFRA persists (no retry or catch-up for failed deliveries) → P-11 | F/services/webhookService.js:89-95 (result.json() inside the try) |
R-16, P-11 |
| Joy email, custom domain | ~18,000/day (19,921 lines in a 27-hour capped sample, 18/11 → 19/11) Custom domain sending failed, falling back to default SMTP: No verified custom domain found on sentemailnotificationv2 |
CONFIG trigger (shop has an activated sender but no verified domain) → CODE persists (the gate checks only activatedSender.isActivated, so every send for such a shop reads the domain, fails and logs) |
Gate F/services/email/sendEmailService.js:129-131; result :209-213; log :162-164, 168-170 |
R-17 |
| Joy email, SMTP PII | Every Message sent via SMTP line logs the subject and the recipient email |
SEC (log line written on every send) | F/services/email/sendEmailService.js:404 |
R-03 |
| Wallet | Apple pass download logs the request headers including authorization: ApplePass … (288 lines in 18 days, on every day). Push-token details are logged (183 lines). |
SEC; confirms the IP-14 remnant, with a second site found at HEAD | Headers F/controllers/loyaltyPassController.js:302-306 (download) and :459-463 (updatable passes); pass/customer dump :345; push token F/services/AppleWallet/ApplePushNotificationService.js:79-82, ~104 |
R-03 (IP-14 remnant) |
Shopify
| Surface | What the logs show | Trigger → why it persists | Evidence in code | Fixed by |
|---|---|---|---|---|
| Shopify Flow | email is a required field 348 (sample, 17 days) on apishopifyflowv2 |
CONFIG (workflow sends empty email) → UX (no guidance) + CODE (200 on failure) | F/controllers/shopifyFlowController.js:275, 342, 500, 1198-1203 |
R-15 |
| Shopify Admin API (throttling) | Sample (cap reached after ~28 hours, 18/11 14:52 → 19/11 18:36 UTC): graphql error … Throttled 12,322 on 28 shops (one shop = 7,747): 8,397 on apihookv2 (inline in webhook handlers), 2,318 on enqueuesubscriberv2, 1,088 on backgroundhandlingorderedv2. The same failures are logged again as Error when get customers shopify 3,044 (all on apihookv2), Error in handleAddOrRemoveTagCustomer Throttled 889 and Error when remove tags customer … Throttled 820 (both on backgroundhandlingorderedv2). ≈ 10.7k throttled calls/day; the customer reads return empty and the tag operations are logged and dropped |
INFRA (per-customer Admin calls inline in webhook paths and on the ordered topic, no cost-aware per-shop queue) → INFRA (no retry: the customer read returns {}, and the dispatcher catches the tag error and returns 'ok'; tag sync silently incomplete) |
Customer read F/services/shopify/shopifyMetaFieldService.js:3453, 3515; tag writes F/services/shopifyService.js:3272-3311, 3775; dispatcher F/handlers/pubsub/subscribeBackgroundHandling.js:1717-1727 |
R-19 |
| Order webhooks → Cloud Tasks | Error enqueueing order webhook to Cloud Tasks: 344 in 18 days (exact, 49 shops): 303 "Task size too large" (9 shops, one shop = 263; 302 orders/updated, 1 orders/create), 32 HTTP 502, 5 HTTP 408, 4 Internal authentication error. The webhook still answers success to Shopify, so Shopify never redelivers: these orders may never have been processed for points |
CODE (the whole ctx.state, shop document plus order payload, is put in the task, :37-41; Cloud Tasks size limit) → INFRA (enqueue failure is logged and returns success: IP-18) |
F/middleware/webhook/enqueueWebhookToCloudTasks.js:37-41, 46-58 |
IP-18, IP-18b, and the affected-orders check |
No error clusters
profile.update_failed timeouts. Low priority; completeness unproven.Exact or capped-sample counts (shares over the displayed counted fingerprints; Mailchimp is a 10-day sample, Flow a 17-day sample; not an all-errors fleet distribution):
| Class (by why it persists) | Log lines | Share | Apps | Fixed by |
|---|---|---|---|---|
| INFRA: work dropped after rate limit (no durable retry, no bulk path) | 205,112 (172,229 + 22,479 + 10,404) | 61.0% | Omnisend, Klaviyo | P-04, P-04b |
| CODE + INFRA: dead credential retried, no breaker | 61,240 (Klaviyo; 27,166 failed refreshes) | 18.2% | Klaviyo | R-02, P-05, P-09 |
| CODE: self-inflicted probe from the admin | 40,714 (25,438 + 15,276; 1,192 of the Klaviyo lines come from the sync path) | 12.1% | Klaviyo, Omnisend | R-01 |
| SEC: secrets/PII in logs | 21,382 (Mailchimp sample) | 6.4% | Mailchimp | R-03 |
| Klaviyo 429 lines themselves | 6,241 | 1.9% | Klaviyo | P-04b |
| CONFIG → UX: misconfiguration nobody sees | 1,674 (1,326 + 348; 663 Klaviyo events) | 0.5% | Klaviyo, Flow | P-05, P-09, R-15 |
| Total | 336,363 | 100% |
Sampled or approximate, not in the shares above (different windows):
| Class | Volume | Apps | Fixed by |
|---|---|---|---|
| CODE: false failure (2xx with non-JSON body counted as failure) | 10,491 in an ~11-day capped sample | Outbound merchant webhooks | R-16 |
| PARTNER → INFRA: merchant endpoint error page or network error, no retry | ~8,600 in the same sample (20,057 − 10,491 − ~920 other plain-text replies) | Outbound merchant webhooks | P-11 |
| CONFIG → CODE: work attempted on every call though the config can never succeed | ~18,000/day (27-hour sample) | Joy email (custom domain) | R-17 |
| CODE + INFRA: dead credential retried (samples) | Mailchimp ~17.4k (401 + 403) / 10 days; Gorgias ~8.9k error lines/day; Instagram ~6.3k / 14 days | Mailchimp, Gorgias, Instagram | R-05, R-14, P-05 |
| SEC: secrets/PII in logs | Wallet auth headers 288 / 18 days; SMTP recipient/subject on every send; Recharge, Omnisend PII | Wallet, Joy email, Recharge, Omnisend | R-03 |
| INFRA: Shopify Admin throttling, no cost-aware queue | ~10.7k throttled calls/day (28-hour sample, 28 shops) | Shopify Admin API | R-19 |
| CODE → INFRA: order webhook not enqueued, success returned | 344 / 18 days (303 "Task size too large") | Shopify order webhooks | IP-18, IP-18b |
| Working as intended | Loox invalid signatures 17 / 18 days | Loox | — |
The five cause types, read across both tables:
invalid_grant branch behind 61,240 lines, the non-JSON 2xx (10,491) and the custom-domain attempt (about 18,000 a day).Not verified: which production build ran on 13/11; whether Omnisend downgrades consent; merchant demand for the new apps; the complete Stamped and Wishlist pulls. From the partner documentation, also not verifiable: Sendlane and PushOwl limits and error codes; whether Klaviyo refresh tokens rotate; whether Postscript external_id and Attentive externalEventId dedupe; the retirement date of Omnisend /v5; what Gorgias means by invalid_client; what Tapcart's customerSession carries; Loop's listing process; Review Rocket's webhooks.
flowchart LR
subgraph PRODUCERS["Producers"]
EV["Loyalty events, customer changes, crons"]
UI["Admin UI"]
end
REG[("Integration registry<br/>browser-safe metadata plus server adapters")]
EV --> ACC[("Durable acceptance<br/>pending delivery record with a logical deliveryId")]
ACC --> DW["Delivery worker on a dedicated Cloud Tasks queue<br/>per-shop admission, attempt id differs from deliveryId"]
SW["Sweeper"] -.->|"repairs failed handoffs"| ACC
DW --> AD["Outbound adapters<br/>Omnisend, Klaviyo, Postscript<br/>each owns retry timing, payload limit and dedupe field, if the partner has one"]
AD --> PARTNER[("Partner APIs")]
DW --> OUT[("Outcome: success, retryable,<br/>or ambiguous to quarantine")]
IN["Partner webhooks"] --> VER["Per-provider verifier<br/>tenant binding, dual-accept window"]
VER --> INH["Inbound handlers"]
AD --> HS["Health service<br/>per app, fenced by installation generation"]
INH --> HS
HS --> DOC[("integration doc health<br/>connected, degraded, auth_broken, capability blocks")]
DOC --> UI
DOC --> FLAG[("shop flags as a derived cache<br/>plus a repair script")]
REG -.-> DW
REG -.-> VER
REG -.-> HS
REG -.-> UI
PAPI["Joy partner API F1 on REST v2<br/>default-deny scopes, quota per shop and partner"] --> EXT["Loop, Tapcart, other partners"]
Integration registry. One code module declares every app: its id, the legacy ids stored in Firestore, its category, the shop flag it owns, its event list and its credential fields. Browser-safe metadata is kept apart from the server adapters. Each entry names its connection source or says "not trackable" (link-only cards). A drift test fails the build when the registry and the real writers disagree. No stored id, flag, route or payload changes shape; the registry aliases the legacy names.
Delivery worker. One dedicated Cloud Tasks queue carries every outbound delivery. A delivery is built from one envelope: schema version, logical deliveryId, attempt id, installation generation, operation, source event id and time, an immutable payload or a reference to it, expiry and status. The worker:
Retry-After rule, and falls back to a jittered backoff when the partner gives no signal;deliveryId into the partner's dedupe field where the partner has one that works;Eight rules come from the partners' documentation and hold for every adapter:
Retry-After header on a 429 (an integer number of seconds). Omnisend returns a retryAfter field in the 429 body on some endpoints and no header; its window slides, so there is no reset moment. Intercom returns X-RateLimit-Reset as a Unix time, and not on every 429. Gorgias returns Retry-after. Attentive, Postscript, LINE and Drip document no wait at all. Drip's window is one hour.unique_id dedupes per profile and metric, so the deliveryId must be fixed before the first attempt and the metric name must not change between attempts. LINE X-Line-Retry-Key dedupes for 24 hours. Postscript external_id and Attentive externalEventId exist, but neither partner documents that it dedupes on them. Omnisend eventID is documented as not working for events that trigger automations. Mailchimp, Drip, Intercom, Sendlane and PushOwl have none. For every partner outside the first two, an ambiguous send is quarantined, or the call must be a state update that is safe to repeat (a Mailchimp member upsert by subscriber hash, a Drip subscriber update by email). A state update still needs "latest state wins", because a retried older update can overwrite a newer one.deliveryId is a UUID. Omnisend eventID and LINE X-Line-Retry-Key accept nothing else.scheduleTime after the adapter's wait, records the deferral, and answers 2xx. A deferral cannot be scheduled more than 30 days ahead. A new queue ramps up slowly, so the first cut-over must not start with a backlog.Replay has an age limit per partner: Attentive does not start a Journey for an event older than 12 hours, and a LINE retry key is valid for 24 hours. "Retried until delivered" always means "inside that partner's limit" (decision 5 in section 8).
Producers keep using the existing topics for the fan-out decision. Only the delivery has to be durable, so the 212-case dispatcher never needs retry: true.
Health service. Each integration document gets a health map: connected, degraded, auth_broken, or disconnected, plus capability blocks for a missing scope. auth_broken is non-destructive: no revoke, tokens kept, delivery paused, merchant told once. Only a merchant disconnect revokes. Three classification rules come from the partners' documentation. First, a 403 is not a 401: Klaviyo permission_denied (a missing scope), a Mailchimp 403 for a user level that does not allow the endpoint, and Intercom api_plan_restricted are capability or plan blocks, never auth_broken and never a reason to refresh a token. Second, a 429 or a 5xx from a token endpoint is transient and never a dead grant; the refresh fence is therefore also a rate fence (Klaviyo allows 10 refreshes a minute). Third, a 410 from Omnisend means the API version was retired: that is "Joy must upgrade", not a merchant problem. The health state becomes the source of truth for an app only after every writer of that app (section 2.1) is migrated and fenced by installation generation; until then the app keeps its legacy truth and shows observational health. The shop flags stay as a derived cache, with a repair script for rollback.
Verifiers. One contract per inbound provider: verify the raw bytes, know where the secret came from, bind the request to an authenticated tenant, dedupe with the event lifecycle in mind, and accept the work durably before acknowledging. Each provider moves through a dual-accept window with per-shop counts of unsigned arrivals; the old registration is removed only after the new one is proven. The check is written per provider, because the schemes differ: Recharge is a plain SHA-256 of the API client secret joined to the body, not a keyed HMAC; Fera is a JWT in the URL; Loop is a base64 HMAC; Judge.me is a hex HMAC whose secret is Joy's one OAuth app secret, so the signature proves the sender and not the shop. Each enforcement window must be shorter than the time the partner waits before it gives up or removes the subscription: Shopify deletes an Admin-API subscription after 8 consecutive failures over 4 hours; Recharge deletes a webhook after 20 failures over 48 hours; Klaviyo disables a webhook that stays in error for more than 48 hours; Yotpo retries at least 10 times and only on a non-200; Loop makes six attempts and never retries a 401.
Credentials by reference. Messages and tasks carry an integration id, never a token. The worker loads the credential after checking shop, app and installation generation, so old queued work cannot run under a new installation. Secret Manager is used only for new app-level secrets. New partner keys are stored encrypted with a real key.
Partner API F1. Partners authenticate with a merchant-issued, per-partner, scoped key on REST v2. The list of permitted routes is default-deny. Quota is keyed by shop and partner with an aggregate shop cap; idempotency is namespaced by partner; a key can be revoked. Partner keys are a new principal type in one shared resolver, not a parallel middleware. OAuth comes later, only for a partner that requires it.
Success counters. Unsampled aggregate counters (source, accepted by Joy, accepted by the partner, delivered where the partner reports it, terminal, quarantined) per operation; diagnostics may be sampled. The legacy operation is measured before each cut-over. "Duplicate" means a duplicate external side effect, not a repeated attempt. Health and retention claims cover instrumented apps only.
deliveryId is separate from the physical attempt task id. A task name only dedupes the enqueue (its reuse is blocked for up to 24 hours) and a task can run more than once. The deliveryId is a UUID. A partner 429 becomes a new task and a 2xx answer to Cloud Tasks, never a 429 answer (rule 8 in section 5.2).
Rejected: retry: true on backgroundHandling*, because the dispatcher swallows errors and a retry would replay hundreds of unrelated, non-idempotent cases. Also rejected: "idempotency = task name".health is the source of truth per app, and only after every writer of that app is migrated and fenced.
Rejected: one global writer on day one, because seven writers stay live. Also rejected: deleting the shop flags (42 or more readers for Klaviyo alone) and live partner checks on read (the source of the self-inflicted 401 lines).secretKey or a constant partner key, which gives full access with no consent and no per-partner revocation.auth_broken is non-destructive, and a missing scope is an operation-level block, not auth_broken. The partners' documentation supports this: Klaviyo answers a missing scope with HTTP 403 and code permission_denied, not with a 401.
Rejected: reusing disconnectIntegration, which revokes the grant at the partner and nulls the tokens.Five groups, in order of execution. Every item states what to do, why, where in code, when it is done, and what it depends on. Section 7 says which wave each item is in.
Small, revertible changes and actions that need no deploy. They stop the leaks and protect merchants before the freeze.
1.1 Freeze the baseline and switch on three alerts IP-40 IP-47 P-01
Rate limit exceeded, max retries reached, dropping task); (2) Unauthorized Klaviyo access plus Klaviyo refresh failures; (3) Shopify order-webhook enqueue failures. No deploy. From now on the non-secret fingerprint strings stay unchanged; a structured field is added next to them instead of rewording.F/services/integrate/syncOmnisendService.js:396, F/handlers/schedule/enqueueHandler.js:544-549, F/services/klaviyoService.js:349-354 and F/middleware/webhook/enqueueWebhookToCloudTasks.js:46-58.1.2 Remove secrets and personal data from integration logs R-03 IP-15 IP-14
authorization header (10-day sample); 288 wallet lines carry authorization: ApplePass; every SMTP send logs a recipient.F/services/integrate/mailchimpService.js:196; F/services/integrate/rechargeService.js:281, F/services/integrate/rechargeService.js:296; F/services/gorgiasService.js:119; F/services/reviews/judge.js:75; F/services/klaviyoService.js:424; F/services/integrate/omnisendService.js:96, :150, :183, :267; F/services/email/sendEmailService.js:404; F/controllers/loyaltyPassController.js:302-306, :345, :459-463; F/services/AppleWallet/ApplePushNotificationService.js:79-82.authorization header, a Recharge order JSON, an ApplePass header or an SMTP recipient.1.3 Stop the admin customer page from probing apps that are not connected R-01 P-03
disconnectIntegration.A/pages/Customer/Details/IntegrationStatus.js:46-54; F/helpers/integration/klaviyo/checkKlaviyoSyncStatus.js:15-27; F/helpers/integration/getStatusSyncIntegration.js:19-26; F/services/klaviyoService.js:349-354.Unauthorized Klaviyo access is at least 80% below the baseline of 1,413 lines a day by 25/11.1.4 Enforce review-webhook signatures where a secret exists, and count the rest IP-08
F/middleware/webhook/verifyHookLoox.js:70-84; F/middleware/webhook/verifyKlaviyoWebhook.js:108-109, :124.1.5 Classify dead Klaviyo and Gorgias credentials, log-only R-02 R-05
invalid_grant answer from the HTTP error and log a typed classification; make the Gorgias refresh return a typed error instead of undefined. No state changes yet. For Klaviyo, classify only HTTP 400 with error: invalid_grant as a dead grant; a 429 or a 5xx from /oauth/token is transient. Klaviyo's token endpoint allows 10 refreshes a minute, and Joy refreshes on every 401 or 403 with no lock, so a 429 there is expected.F/services/klaviyoService.js:717-758; F/services/gorgiasService.js:198-238.Cannot destructure property 'accessToken' no longer appears.1.6 Check whether Omnisend contact sync downgrades consent R-18
sendWelcomeMessage column (it defaults to true). Run the matrix on the API version the new adapters will use (2026-03-15), not only on /v5. If a downgrade is confirmed, omit channels on update (a one-line, revertible fix).nonSubscribed. If Omnisend applies it to existing contacts, each sync shrinks the merchant's audience, and every bulk path multiplies the harm. Omnisend's documentation does not say whether a sent status overrides the stored one, so only the sandbox can answer. The documented upsert is keyed on the email identifier, and Joy also sends phone-only contacts.F/services/integrate/omnisendService.js:221-231.1.7 Find the orders that lost points (ops, no tag)
F/middleware/webhook/enqueueWebhookToCloudTasks.js:46-58; retention in F/helpers/utils/getTimeToLiveExpire.js.1.8 Contact shops with dead credentials (support, no tag)
1.9 Take down the public Tapcart guide N-06a
secretKey.ROOT/docs/design/features/feat-weaverse-hydrogen-blocks.md:523-531.1.10 Label integration tickets and freeze the ticket baseline IP-80
IP-50) using the log fingerprints of section 4.One list per integration. Items that say "through the health service" need group 3 item 3.6.
2.1 Mark a revoked Klaviyo grant as broken instead of retrying it R-02
invalid_grant branch. Treat only HTTP 400 with error: invalid_grant as a dead grant; a 429 or a 5xx from /oauth/token is transient and must never set auth_broken. On invalid_grant, re-read the document; if the stored refresh token differs from the one that just failed (a reconnect happened while old work was in flight), retry once with the stored one. Otherwise mark the integration auth_broken through the health service. No revoke call: the grant is already invalid at Klaviyo. Do not null the tokens; they cannot be reused and are kept only for diagnosis, so recovery is a merchant re-authorisation. On a 200 refresh, store refresh_token as well when the response returns one. Serialise refreshes per shop: the token endpoint allows 10 refreshes a minute. Send a 403 permission_denied to the capability block, never to a refresh.invalid_grant on refresh means the refresh token is invalid, and that a refresh token unused for 90 days is revoked. A concurrent refresh by another worker does not invalidate the refresh token, so the re-read guards against a reconnect, not against a refresh race.F/services/klaviyoService.js:717-758 (the revoke and token null at :732-748; only accessToken is stored on success, at :722-728); the refresh on every 401 or 403 at F/services/klaviyoService.js:328-345.auth_broken once, its refresh attempts stop, and its tokens are still on the document; a forced 429 from the token endpoint leaves the state unchanged.2.2 Fix the two known Klaviyo retry bugs R-06 IP-23
privateToken (it reads two field names that do not exist); add the missing handler for checkKlaviyoBulkImportJobs.F/handlers/schedule/enqueueHandler.js:568-569; enqueued at F/services/integrateKlaviyoService.js:116, missing from the map at F/handlers/schedule/enqueueHandler.js:165-212.checkKlaviyoBulkImportJobs no longer appears.2.3 Send the Klaviyo profile properties merchants segment on R-12
lists:write scope as well as profiles:write, so a private key without lists:write fails on the bulk path while the single-profile path works: report that as a capability block. Update the Klaviyo Marketplace listing copy.F/services/klaviyoService.js:466-492.2.4 Keep events on after an Omnisend reconnect R-09
isEnabledOmnisendTriggerEvent from the length of omnisendSubscribedFields, as the save path already does..length of a boolean, so the flag becomes false and events stop until the merchant saves the settings again.F/controllers/integrationController.js:740 (the correct form is at :480).The Omnisend drops (172,229 contact updates, 10,404 events) are fixed by the delivery worker (3.4) and the bulk path (3.5), after the consent verdict (1.6).
Three facts about Omnisend's API change that work:
/v5 with a date-versioned API (the Omnisend-Version: 2026-03-15 header, base URL https://api.omnisend.com/api/) and publishes a v5 migration guide. Joy still calls /v5 (F/services/integrate/omnisendService.js:9). No v5 retirement date is published. The new adapters (3.4 slices 1 and 2, and 3.5) are built against 2026-03-15 so that the work is not done twice (checked when the item is built). A retired version answers 410.Retry-After header. The limit is 400 requests a minute per brand, across all endpoints and all integrations of that brand, on a sliding window. Joy's events and contacts therefore compete in one budget, and Joy shares that budget with the merchant's other apps.2.5 Isolate shops in the Mailchimp cron and fail stuck jobs R-08 IP-24
isSyncing.isSyncing blocks that shop's per-customer sync for good.F/handlers/schedule/syncMailchimpCustomers.js:17-55.operation_id is not enforced as unique; batch results can be downloaded for seven days.Mailchimp's dead credentials (about 17,400 lines of 401 and 403 in 10 days) are handled through the health service (3.6), which states how Mailchimp's 401, 403 and 429 are read.
Check during the spike: the per-customer path adds every synced customer who is new to the list as subscribed, with no marketing-consent check (F/services/integrate/mailchimpService.js:81-83). This is the same class of problem as item 2.10. v8 adds no item for it; confirm with product whether it is intended.
2.6 Mark a failed Gorgias refresh as broken, and find out why it started R-05
auth_broken through the health service. Separately, check whether Joy's own Gorgias OAuth client is still valid.invalid_client on 8 accounts since 2026-11-05 01:46 UTC, none before. Gorgias does not document this error; in OAuth 2.0 it means client authentication failed. Joy sends the same client id and secret for every account and the token endpoint is per helpdesk subdomain, so a dead Joy client would fail every account, not 8 of 58. Test account-level causes first (the app was uninstalled or deactivated on that helpdesk, the subdomain changed, the account closed) by refreshing one healthy account and comparing. Gorgias documents that an access token lasts 24 hours and that a deactivated app needs the whole authorisation flow again.F/services/gorgiasService.js:198-238.auth_broken once and the error lines stop; the cause of the 05/11 change is written down.2.7 Show an expired Instagram session and ask for a reconnect R-14
auth_broken through the health service and show a reconnect prompt. Add a scheduled refresh for tokens that are more than 24 hours old and have less than about 30 days left, so that an idle shop does not reach the expiry. Treat Meta error 190 (subcodes 460, 463 and 467) as auth_broken, and codes 4 and 17 (too many calls) as transient.auth_broken reports the problem without preventing it.F/services/instagram/instagramApiService.js:41-47, F/services/instagram/instagramApiService.js:63-77 (the 24-hour window), F/services/instagram/instagramApiService.js:103-105.2.8 Rotate the Instagram app secret IP-05
INSTAGRAM_APP_SECRET and INSTAGRAM_REDIRECT_URI; then rotate the secret and remove the constant fallbacks.F/config/instagram.js:5-9.2.9 Stop Drip from looking disconnected after a transient error R-13
statusConnectedDrip to false, while the integration document still says connected.F/services/dripService.js:371-378 (bulk) and F/services/dripService.js:244-245 (per customer).2.10 Subscribe only customers who consented R-11 IP-29
F/services/integrateAttentiveService.js:435-449; the subscribe call at F/services/attentiveService.js:126-158.2.11 Backfill signature keys and tell Loox merchants IP-08
F/repositories/integrationsRepository.js:80-82; F/middleware/webhook/verifyHookLoox.js:70-84.2.12 Replace the Chatty shared constant key IP-03 IP-05
F/const/integration/chatty.js:1; F/middleware/webhook/verifyOAuthHook.js:2-4; F/services/integrate/joyWebhookService.js:32.2.13 Make failing Flow actions fail visibly R-15
message key when an action fails, as the reward-by-program action already does; Flow shows the raw response to the merchant and does not resend. Do not use 5xx, 429 or 202, and do not take longer than 10 seconds: Flow resends those for up to 36 hours, and the middleware would answer the resend as a duplicate with 200. Move the action_run_id marker to after the handler completes (the acceptance contract of 2.19), so that a resend after a timeout is processed instead of acknowledged. A bad HMAC gets 401, compared timing-safe.success: false, so Flow records a success when no points were given (348 email is a required field lines in a 17-day sample). The same trap exists on two more paths than v7 named: an action slower than 10 seconds and an action that answers 429 are both resent by Flow and both acknowledged as duplicates.F/controllers/shopifyFlowController.js:275, :342, :500, :1198-1203; the correct pattern is at :1156-1159; the marker and the HMAC check at F/middleware/webhook/verifyHookShopifyFlow.js:26-62; the 10-second function timeout at F/entries/http-integrations.js:24-27.2.14 Count any 2xx answer as a delivered webhook R-16
ok.F/services/webhookService.js:89-95.send hook error lines from 2xx answers are zero by 18/12.2.15 Block webhook URLs and SMTP hosts that point inside the network IP-12
F/services/webhookService.js:106-121; F/controllers/apiHookV1/webhookController.js:33-39; F/controllers/customSenderEmailController.js:289-310; F/services/email/sendEmailService.js:471-485.Retry and catch-up for failed webhooks is item 3.12.
2.16 Try the custom domain only when one is verified R-17
F/services/email/sendEmailService.js:129-131; the log at :162-164 and :168-170.2.17 Re-encrypt merchant SMTP passwords with a real key IP-06
F/const/email/keySmtp.js:1; F/services/email/sendEmailService.js:471-485; F/controllers/customSenderEmailController.js:289-310.2.18 Stop losing throttled Admin API calls R-19
errors[].extensions.code equal to THROTTLED on an HTTP 200 answer, not from the status. The calls move onto a cost-aware per-shop queue that admits work against extensions.cost.throttleStatus and waits at least one second after a throttle. Mass tag runs go through a bulk mutation built on tagsAdd and tagsRemove. The existing bulk tag code must not be reused as it is: it sends customerUpdate with the full tag array, which overwrites a tag the merchant changed in between. The remaining operations are sized after this slice.F/services/shopify/shopifyMetaFieldService.js:3453, F/services/shopify/shopifyMetaFieldService.js:3515-3519; F/services/shopifyService.js:3302-3310 (the per-customer tagsAdd and tagsRemove mutations are at :3705 and :3750); F/handlers/pubsub/subscribeBackgroundHandling.js:1717-1727; the bulk tag code that uses customerUpdate is F/services/shopify/bulkMetafieldService.js:471-477.Throttled lines for those operations fall against the baseline.2.19 Accept order webhooks durably, by reference IP-18b
X-Shopify-Webhook-Id (the delivery), as Shopify documents; keep X-Shopify-Event-Id only for correlation. Before that change, count in the shadow mode of 2.20 how many deliveries are dropped today because they share an event id with a different webhook id or topic. Add a sweeper that repairs failed queue handoffs. Enqueue a small reference task instead of the whole request state.F/middleware/webhook/enqueueWebhookToCloudTasks.js:37-41; F/middleware/webhook/verifyHook.js:141-150; F/repositories/webhookLogRepository.js:20-37.2.20 Answer Shopify honestly when a webhook is not accepted IP-18
ALREADY_EXISTS for the same payload counts as success. Verify against the raw body only and compare timing-safe; drop the second accepted form (the HMAC of the re-serialised JSON) after the shadow week shows nothing depends on it. Add a non-2xx rate alert and a daily check, through the server path, that the webhook subscriptions still exist.F/services/shopify/graphqlService.js:2102-2103). So enforcement needs the shadow week first. The 401 for a bad HMAC is also an App Store requirement for the mandatory compliance webhooks: an app that answers them otherwise is rejected in review. That makes this item a compliance item as well as a reliability one.F/middleware/webhook/verifyHook.js:40-52 (the two accepted forms), F/middleware/webhook/verifyHook.js:135-140; F/middleware/webhook/enqueueWebhookToCloudTasks.js:46-58.orders/* subscription is intact.customers/data_request, customers/redact, shop/redact) pass through this verifier? The app configuration file is not tracked in the repository, so the route could not be confirmed.2.21 Fix the three Joy Subscription problems R-21
Note on the tag: v6 used R-20 for this item in one place and for the re-sync in another. From v7 on, R-21 is Joy Subscription and R-20 is only the re-sync (item 2.22).
subscription_billing_attempts/failure subscription at the sender once both app owners agree; during the rollout answer it with an authenticated no-op. A failed billing attempt is not a reason to change loyalty state.F/controllers/joySubWebhookController.js:84; F/middleware/webhook/verifyJoySubWebhook.js:63-91.2.22 Re-sync current state for shops that lost deliveries R-20
F/services/triggerIntegrationAppService.js:381-445.Each step ships alone and is switched on per app and operation behind a gate. It is proven on one app first: Omnisend events for delivery, Klaviyo for health.
3.1 Outcome log for every integration call P-01 IP-41
F/services/reliableJobs/runReliableJob.js:36-53; first call sites are the drop points at F/services/integrateKlaviyoService.js:779-781, F/services/integrate/syncOmnisendService.js:507-510 and F/services/triggerIntegrationAppService.js:209-213.3.2 Integration registry P-02
F/const/programs/integrationPrograms.js:8-15 and the maps at A/const/integrationStatusFields.js:27-47.3.3 Read paths stop writing state P-03
F/controllers/integrationController.js:322-341.3.4 Delivery worker P-04 IP-20 IP-22 IP-26
eventID (the deliveryId, a UUID) and reuse the original eventTime on every attempt, for the historical-dedupe case, but do not count on it. Dedupe is on Joy's side only. An Omnisend event whose send timed out is ambiguous and goes to quarantine, exactly like a send that triggers an SMS. The answer is 202, so acceptance is asynchronous and every timeout is ambiguous. On a 429, read retryAfter (seconds) from the JSON body when it is present and otherwise back off with jitter. Keep Joy's admission per shop well below 400 requests a minute, in one budget for events and contacts. Build against API version 2026-03-15 (checked when the item is built).unique_id, so the deliveryId is fixed before the first attempt and the metric name, which includes the event prefix, does not change between attempts. Klaviyo's limits are per endpoint tier (Create Event 350 a second and 3,500 a minute; Bulk Import Profiles 10 a second and 150 a minute), so a 429 on bulk import must not pause events; one throttle flag per shop is too coarse. A private-key shop shares its quota with the merchant's other private-key integrations.externalEventId equal to the deliveryId, but treat dedupe as unproven until the contract test shows it. An event delivered more than 12 hours after occurredAt does not start a Journey, so the replay-age limit for Attentive events is 12 hours; a late delivery must not rewrite occurredAt to "now" without a product decision. Attentive sends no Retry-After header: back off with jitter.F/const/pointExpiration.js:39-43; enqueue through F/services/cloudTaskService.js:25-52; first producers at F/services/integrate/syncOmnisendService.js:456-510 and F/services/integrateKlaviyoService.js:737-781. Today's Omnisend event is sent with no eventID and a new eventTime on every attempt (F/services/integrate/omnisendService.js:173-181); today's Attentive event is sent with no externalEventId (F/services/attentiveService.js:243-254).deliveryIds and the oldest pending delivery is under the agreed age (by 18/12); then the same for each later adapter. "No duplicate deliveryId" measures Joy's attempts, not Omnisend's side effects; ambiguous Omnisend events are counted as quarantined, not as delivered./v5 retirement date).3.5 Bulk path for mass operations P-04b
klaviyoSync producers onto them. Omnisend batches only after the consent verdict. The partner limits that size the batches: a Klaviyo bulk import takes up to 10,000 profiles and 5 MB a request, at 10 requests a second and 150 a minute; an Omnisend batch takes up to 100 items, and both the batch requests and the status polls draw on the same 400 requests a minute per brand as every other Omnisend call. A batch is accepted with a job id; completion and per-item errors must be read back.F/handlers/pubsub/subscribeBackgroundHandling.js:2563-2564; F/services/integrateKlaviyoService.js:991-1030; F/handlers/schedule/enqueueHandler.js:544-549.dropping task line.3.6 Health service, per app P-05 IP-46
auth_broken state and the capability block; add the refresh fence; run one week log-only and measure false positives; then enforce, with a reconnect that reconciles current state (no replay of time-bound events) and a flag-repair script for rollback. Klaviyo first, then Gorgias, Instagram and Mailchimp. Other apps stay observational.permission_denied is a capability block: match on the status and the code and parse the scope list, so that profiles:write and lists:write are covered as well as events:write. Today the code matches one English sentence for one scope, on the private-key path only.auth_broken with no retry. A 403 has three meanings (the authorising user lost access, the user's level does not allow the endpoint, the account is deactivated); read title and detail before choosing auth_broken or a capability block. A 429 is a concurrency limit and carries no Retry-After.auth_broken instead of isConnected: false, and a 403 api_plan_restricted is a capability block; today a 403 falls to the generic throw (F/services/intercomService.js:137-140).F/services/email/sendEmailService.js:429-462; the destructive path to replace is F/repositories/integrationsRepository.js:899-933; the Klaviyo scope match is F/services/klaviyoService.js:347-351.3.7 Replay and quarantine for operators P-06 IP-19
ROOT/scripts/setup-pubsub-ordering.sh:136; F/handlers/pubsub/subscribeOrderedDlq.js.3.8 Inbound verifiers per provider P-07 IP-07 IP-08 IP-09
JUDGEME-HMAC-SHA256 header, a hex HMAC-SHA256 of the raw body. Joy registers the webhook through its OAuth app, so the secret is Joy's one Judge.me client secret for every shop. The signature proves the sender is Judge.me, not which shop the review belongs to; the tenant must still be bound from the payload.jwt URL parameter. It is HS256, signed with the store's Fera secret key, which Joy already stores. Check exp (the token lasts one hour) and that store_domain matches the tenant. No URL token and no merchant action are needed.X-Recharge-Hmac-Sha256 as the hex SHA-256 of the API client secret joined to the raw body; this is not a keyed HMAC, so a verifier written like Joy's others would reject every genuine delivery. The API client secret is not the API token Joy stores: collect it at connect, run dual-accept until each shop has supplied it, and take the tenant from the verified credential instead of the shopId query (checked when the item is built). The API token needs read_orders for order/created.WebhookMessageIdPerTarget header.F/middleware/webhook/verifyHookReviewApp.js:17-57; F/middleware/webhook/verifyKlaviyoWebhook.js:108-109; F/middleware/webhook/verifyHookLoox.js:70-84; F/routes/apiHookV1.js:627; the Recharge handler at F/controllers/integrationController.js:769-778 and the webhook registration with the API token at F/services/integrate/rechargeService.js:92-108; the stored Fera secret key at F/services/reviews/fera.js:15-17.3.9 One auth contract for OAuth apps P-08 IP-10
state for Klaviyo, Attentive, Mailchimp, Omnisend, Judge.me, Drip and Gorgias; one revoke path; a partnerToken auth kind for partners such as Postscript.state; the others pass a raw shop id or domain.F/helpers/social/oauthState.js:16; F/controllers/integrate/klaviyoController.js:23; F/controllers/integrate/attentiveController.js:19; F/controllers/integrationController.js:226.state.3.10 Merchant health UI P-09 IP-51
/translate.A/pages/Integration/Integration.js:62-79; A/components/BannerNotice/BannerIntegrationDisconnected.js:30; A/const/integrationDisconnectConfig.js:18-25.degraded visible to merchants).3.11 Move the remaining profile syncs onto the worker P-10 R-10 IP-25
syncCustomerTo* publishers out of the customer repository; queue changes made during a bulk sync instead of dropping them; keep legacy handlers until their queues drain.X-Line-Retry-Key equal to the deliveryId (it must be a UUID), identical on every attempt of the same message and recipient. Retry only on a 5xx and on a timeout, within 24 hours of the first attempt. Treat 409 as success: LINE answers 409 when it had already accepted the request. Do not blind-retry a 4xx. For a 429, read the error message: a rate-limit 429 may be retried with backoff, while a monthly-quota or per-user 429 is terminal until the quota changes and shows in health as a capability block. The push limit is 2,000 requests a second per channel.F/services/gorgiasService.js:26-32), so with several instances the real rate is unbounded. Admission uses the account as the limit scope and honours Retry-after.X-RateLimit-Reset is correct and is kept.isSyncing is true is dropped, which is the source of "profile not updated" tickets.F/repositories/customerRepository.js:7224-7440 (the skips at :7236, :7265, :7301); F/helpers/klaviyoTaskQueue.js:27-50.3.12 Retry and catch-up for outbound merchant webhooks P-11 IP-21
merchant_webhook adapter on the worker: retry with backoff for 5xx and timeouts, a catch-up sweeper, and "send, then mark as sent". The shop secret leaves the message payload.F/handlers/onCreateActivity.js:81-84; F/services/webhookService.js:89-95; F/handlers/schedule/frequentOperations.js:14-18.3.13 Contract tests and a staging sandbox P-13 IP-28
invalid_grant) and a staging sandbox per partner. Each earlier cut-over already carries its own failure tests. The sandbox is where the undocumented dedupe fields are proven: Attentive externalEventId and Postscript external_id.F/services/integrate/__tests__/klaviyoSampleContract.test.js.3.14 Instrument the remaining outbound and inbound paths P-14b IP-43 IP-44
F/middleware/webhook/verifyHook.js:135-140; F/middleware/webhook/enqueueWebhookToCloudTasks.js:46-58; F/middleware/webhook/verifyHookReviewApp.js:17-57.3.15 The remaining alerts P-14c IP-47
IP-51).F/entries/schedule.js.3.16 Integration diagnostics for support P-14d IP-49
F/routes/api.js:870-873; pages under A/pages/DevZone/.Deferred to next quarter: registry-driven dispatch (P-12); the BigQuery sink and hourly rollup (P-14a, IP-42), with the BigQuery part of the diagnostics page (IP-49); instrumentation of the remaining surfaces (IP-45); the fleet dashboard (IP-48) and the weekly report that needs it (IP-82); URL tokens for Yotpo and Review Rocket (Fera is verified through its JWT in 3.8).
Why these apps and what competitors do is in research-new-integrations.md and research-potential-apps.md. In short:
| App | Competitors with a documented integration (of 9) | What Joy copies or does better |
|---|---|---|
| Postscript | 6: Smile, LoyaltyLion, Yotpo, Rivo, Gameball, Okendo Loyalty | Push model like Rivo and Okendo: properties plus events, existing subscribers only, never subscribe anyone |
| Loop Returns | 3: Smile, Yotpo, Rivo (all built by Loop, September 2026) | Joy can lead: no competitor awards points when a customer picks an exchange over a refund |
| Tapcart | 5: Smile, LoyaltyLion, Yotpo, Rivo, Okendo Loyalty | Match the in-app block; fix the key exposure first |
| Okendo and other review apps | 3 plus Okendo's own product | The Flow template is the market-proven route (Rivo uses it) |
4.1 Start the partner applications and measure demand (business development, no tag)
4.2 Flow review templates and the plan-gate fix N-01
flow_template extensions (Okendo, REVIEWS.io, Junip, Growave) that call the reward-by-program action; a Junip card; and the plan-gate fix: return 402 with an upgrade message instead of an empty 404.F/controllers/shopifyFlowController.js:135-143; the template to copy is ROOT/extensions/flow-template-loox/shopify.extension.toml; cards at A/config/integration/appList.js:567-616.action_run_id credits nothing; the templates are submitted to Shopify (by 18/12).rating and its guide states that a review rejected later has already been rewarded. Okendo itself documents awarding a third-party loyalty app's points through this trigger. The 402 is safe for Flow: Flow does not resend a 4xx other than 429 and shows its message to the merchant.4.3 Tapcart secure block N-06a
customerSession carries; then a Joy custom block that gets a token from the headless API with a verified customer-session identity and never holds a secret; a rewritten guide; and per-shop key rotation after the new block is live.F/controllers/headless/headlessAuthController.js:33-36; F/services/headless/headlessAuthService.js:20-27.customerSession that Joy's server can verify with Shopify without a secret in the block? Tapcart documents it only as "authentication tokens and session data", present when the customer is logged in. If the answer is no, the block cannot prove identity and the design changes. (2) Which Tapcart plan custom blocks require (a search result says Enterprise; not confirmed), since it bounds the number of shops the block can reach.4.4 Loop listing N-04a
4.5 Subscription pack, sizing only N-08
4.6 Postscript beta N-03
F/services/integrateAttentiveService.js:49-118; the event list to copy is F/const/integrations.js:338-354; the card logo already exists at A/config/integration/appList.js:141.deliveryId is passed as external_id with durable acceptance (3.4); until the spike has proven that Postscript dedupes on external_id, every ambiguous event send is quarantined, never resent (decision 5); observational health (3.6); the per-shop key is stored with real encryption; the partner token is an app-level secret; Postscript has approved Joy as a partner; the demand check passed; admission stays at or below 7.5 profile updates per second per partner and shop (two calls each, a lookup and then an update by subscriber id, against a limit of 15 requests per second that events also draw on).Retry-After header: back off with jitter on error code 3002. "Never subscribe" holds by construction: the properties and events endpoints cannot create a subscriber or change opt-in status.external_id twice on a test shop and see whether Postscript records one event or two. Its documentation says only that the field is the event's unique id on the sender's system.4.7 Loop exchange bonus N-04b
return topic with the return.closed trigger only, verified with X-Loop-Signature (a base64 HMAC-SHA256 of the raw body with the merchant's Loop webhook secret, which the merchant supplies at connect). It awards points through a merchant-configured program. Loop's payload has no field that says which outcome the customer chose, so the exchange is detected from the payload: award when state is closed and the payload has a non-empty exchanges array or a gift_card amount above zero. Identify the customer by customer (the email) and provider_order_id. Answer within 10 seconds and dedupe by return id, because Loop delivers at least once.return.closed topic; Loop has no such topic. The topic is return and return.closed is one of its triggers; a subscription to both return.updated and return.closed would deliver a close twice.F/routes/apiHookV1.js:627, on the verifier contract (3.8); the task type goes in F/handlers/schedule/enqueueHandler.js:165-212.Deferred: Zendesk sidebar app (N-05: demand unmeasured, Gorgias covers the Shopify-native helpdesk); Tapcart-native support (N-06b: only if Tapcart engages, needs F1); Okendo native connector (N-07: the Flow template is the end state unless Okendo asks); Postscript general availability; the subscription pack build.
Next wave, after 17/02: the subscription pack (Loop Subscriptions, Skio, Stay AI); a returns pack (Redo, AfterShip Returns) that reuses the exchange bonus; a Gladly helpdesk card, only after helpdesk demand is proven; zero-party Flow templates (Kno, Octane AI), each gated on a confirmed Flow trigger. None of these is outbound, so none waits for the delivery worker.
Not picked: Klaviyo SMS (covered by the Klaviyo sync); Yotpo SMS (shut down on 2025-12-31); native REVIEWS.io, Junip and Growave (covered by 4.2); Re:amaze and WhatsApp (on hold); Social Snowball, Aftersell, Smartrr native, ConnectPOS, Zalo ZNS, Vajro, Richpanel, wholesale apps, Rise.ai, Segment, Triple Whale, UpPromote and page builders.
The partner API is listed last because it is the outward-facing contract, but it is built early (Wave 1): the Loop listing and any Tapcart-native work wait for it.
5.1 F1, the partner-ready API N-02 IP-73 IP-76 IP-77
/integrate.F/middleware/rest/verifyAuthen.js:172-251 (a new scope must never fall through the check at :247, which blocks only the read-only scope); plan rules at F/config/subscription/plans.js:893-905.5.2 REST contract test in CI IP-72
customers/migrate paths from the public spec and run the spec contract test in CI.ROOT/joy-api-v2.yaml; ROOT/.gitlab-ci.yml.customers/migrate.5.3 REST v1: measure callers, publish the migration table IP-69 IP-70
Deprecation, Sunset and Link headers to REST v1 answers; measure who still calls v1; publish a v1 to v2 migration table and correct the v1 spec.F/routes/restApiV1.js; F/handlers/restApi.js:34-36; ROOT/joy-api.yaml.5.4 REST v1 sunset: notice now, 410 later IP-71
F/routes/restApiV1.js.5.5 Versioning policy and a channel to reach developers IP-74 IP-79
5.6 Smax: drain the stored batches or remove them IP-61
F/handlers/schedule/processSmaxJobs.js:19-57; F/services/integrate/smaxService.js:274-281.5.7 Label the link-only cards honestly IP-67
A/config/integration/appList.js:187./translate.Deferred to next quarter: the npm publish pipeline (IP-75), the headless SDK contract test (IP-78), the REST v1 410.
| Wave | Dates | What goes in it |
|---|---|---|
| Wave 0, before the freeze | Fri 20/11 → Wed 25/11 | Group 1, all of it: 1.1 to 1.10. Deploys: 1.2, 1.3, 1.4, 1.5 (and the 1.6 fix only if a downgrade is confirmed) |
| Wave 0, BFCM freeze (dates proposed) | Thu 26/11 → Tue 01/12 | Branch work and read-only work, no deploy: the delivery-worker spike (3.4), the first registry entries (3.2), the Klaviyo writer inventory (3.6), the contract-test branch (5.2), the Tapcart session spike (4.3), F1 design, partner applications (4.1) |
| Wave 1 | Wed 02/12 → Fri 18/12 | Week 1: outcome log (3.1), small fixes (2.2, 2.4, 2.5, 2.9, 2.13, 2.14, 2.16), registry and read paths (3.2, 3.3), contract test merged (5.2), key backfill (2.11), Instagram secret (2.8). Week 2: delivery worker slice 1, Omnisend events (3.4). Week 3: Klaviyo health log-only (3.6, 2.1 first step) and order-webhook shadow mode (2.20), both observational. Across the wave: durable order acceptance (2.19), Joy Subscription parts a and b (2.21), Flow templates (4.2), F1 (5.1) |
| Wave 2 | Mon 21/12 → Mon 18/01/2027 | Holiday weeks (21/12, 28/12): branch work and log-only modes. From 04/01: Klaviyo health enforced (3.6, 2.1) with the badge and banner (3.10); order-webhook enforcement (2.20); delivery worker slice 2 (3.4), one adapter a week; bulk path (3.5); replay surface (3.7); verifiers (3.8); SSRF guard (2.15); Attentive consent (2.10); Tapcart secure block (4.3); Chatty key (2.12, if Chatty is ready); Klaviyo profile subset (2.3); health for Gorgias, Instagram and Mailchimp (3.6, 2.6, 2.7); Admin throttling first slice (2.18); Joy Subscription part c (2.21); per-shop re-sync (2.22); Loop listing (4.4); subscription sizing (4.5); Postscript spike and narrow slices (4.6) |
| Wave 3 | Tue 19/01 → Wed 17/02/2027 (no work during the Tết break, 04/02 → 10/02) | Remaining profile adapters (3.11); merchant-webhook retry (3.12); contract tests (3.13); instrumentation, alerts and diagnostics (3.14, 3.15, 3.16); SMTP re-encryption (2.17); auth contract (3.9); REST v1 work and developer channel (5.3, 5.4, 5.5); Smax and link-only cards (5.6, 5.7); Postscript build and beta (4.6); Loop exchange bonus (4.7). The third delivery adapter reaches the end of its soak here |
Rules that hold in every wave
auth_broken banner, the REST v1 headers) ships with a help-centre article, a support macro and one line of in-app text. Tickets may rise in the week after such a change; the ticket trend is read net of that.No production behaviour changes before its gate is met. A gate is a metric that exists before the deploy, a trigger that forces a rollback, and a rollback mechanism.
| Gate | Change it guards | Metric that must exist first | Rollback trigger | Rollback mechanism |
|---|---|---|---|---|
| G-0 | Wave 0 deploys: 1.2, 1.3, 1.4, 1.5 | Alerts 1 and 2 of item 1.1 | Any new error class | Revert tag, under 30 minutes |
| G-1 | Delivery worker slice 1, then each adapter (3.4, 3.11) | Sent, retried, failed and ambiguous counts per app; queue depth; age of the oldest task; duplicate-deliveryId rate (Joy's attempts; a partner without working dedupe is covered by the quarantine count); the count of 429 or 503 answers the worker gave to Cloud Tasks, which must be zero |
Success over eligible logical deliveries below the legacy baseline, or any duplicate | The producer flag routes new work back to the legacy path; accepted work drains on the new path or is quarantined, under 15 minutes; soak of at least 3 days before the next adapter |
| G-2 | Health service, log-only to enforced (3.6) | Would-be transitions per hour; false-positive rate against later successful calls; an alert when more than an agreed number of shops enter auth_broken within an hour |
False-positive rate above the threshold of decision 4 | Flag off, then the flag-repair script (dry-run diff first) |
| G-3 | Order webhooks, shadow to enforced (2.19, 2.20) | Would-be non-2xx per topic for one week; answers slower than 5 seconds per topic; deliveries that share an event id with a different webhook id; non-2xx rate alert; subscription-existence check | Non-2xx rate above the shadow baseline, or any missing orders/* subscription |
Revert tag; re-register subscriptions through the server path |
| G-4 | Verifier enforcement per provider (3.8) | Per-shop counts of unsigned arrivals; authenticated-installation coverage | A legitimate delivery is rejected | Per-provider dual-accept flag. The flag must be turned back inside the partner's own give-up window (section 5.2, Verifiers), or the partner removes the subscription |
| G-5 | SMTP re-encryption (2.17); Tapcart key rotation (4.3) | Decode failures; SMTP failures; list of affected shops | Any spike in decode failures | Read-both stays on; rotation is per shop, after the new block is live |
| G-6 | 2.13, 2.14, 2.16 and the 1.6 fix | Flow run errors; send hook error rate; fallback lines per day; Omnisend contact and consent counts for a pilot shop |
An unexpected rise | Revert tag |
Unauthorized Klaviyo access at least 80% below 1,413 lines a day.deliveryId on Joy's side and every ambiguous send quarantined; the Klaviyo log-only false-positive rate measured; no send hook error from 2xx answers; custom-domain fallback lines down 95%.Only the questions that block work.
| # | Decision | Who decides | By when | What it blocks |
|---|---|---|---|---|
| 1 | Who staffs new-app work? Hire a developer, borrow one from another Joy app for 8 to 10 weeks, or run the reliability-first fallback described below | Winn | Before Wave 1 starts on 02/12 (derived from the wave plan; v6 set no date) | 4.2, 4.4, 4.5, 4.6, 4.7, 5.1, and the Wave 2 slots of 2.18 and 2.21 part c, which v6 gave to this developer |
| 2 | F1 plan and pricing rule, as one pricing page: may partners call on Free; does partner traffic share the merchant's quota; do custom actions and headless stay Ultimate-only | Winn and product | 25/11 | 5.1 (F1), 4.4 and 4.7 (Loop), and Tapcart's reach (4.3, Tapcart-native) |
| 3 | Gorgias OAuth client: is Joy's client still valid, and what changed on 2026-11-05 01:46 UTC when invalid_client began on 8 accounts. The test that answers it: refresh one healthy account with the same client and compare. If the healthy account still refreshes, the cause is at account level and the 8 shops must reinstall the app |
Winn (owner of the Gorgias app credentials) | 25/11, before support contacts the Gorgias shops | 1.8 for Gorgias shops, 2.6 |
| 4 | Health thresholds: the false-positive limit before enforcement; whether Joy ever auto-disconnects (none until decided); whether degraded is visible to merchants |
Winn, with product and support | Before Klaviyo enforcement, the week of 04/01; the log-only week in Wave 1 supplies the numbers | 3.6 enforcement (gate G-2), 2.1, 2.6, 2.7, 3.10 |
| 5 | Ambiguous sends that trigger an SMS or an event: avoid a duplicate, avoid a loss, or hand to an operator to reconcile; and the maximum replay age, one value per partner (Attentive Journeys ignore events older than 12 hours; a LINE retry key lasts 24 hours). From v8 this also covers Omnisend events, which have no partner-side dedupe for automations, and PushOwl pushes | Winn and product | Quarantine is the default for Omnisend events from slice 1 in Wave 1 (week of 07/12). The decision itself is needed before the first adapter for a send that reaches a customer's phone, in Wave 2 | 4.6 (Postscript beta), the quarantine policy in 3.4, 3.7 |
If decision 1 is "nobody": the reliability-first fallback applies. Order-webhook work (2.19, 2.20), Joy Subscription (2.21), Admin throttling (2.18, ahead of 2.3) and the credential and tenant-boundary work keep their waves. F1 shrinks to a read-only pilot for one confirmed consumer. The Flow templates move to Wave 3 or next quarter. There is no Postscript, no Loop exchange bonus and no Klaviyo profile subset this quarter. The safety work (gates, durable acceptance, fences) is never cut.
Not decisions for now (user, 20/11): questions with no error evidence in the 18-day logs are not listed as blocking decisions: Smax; Omnisend API version; Attentive subscribe; Recharge verification secret. They are settled by the owner of the item when it is built (see each item's "Check during the spike" line).
auth_broken is non-destructive, the refresh fence re-reads the stored refresh token before deciding and limits the refresh rate, one log-only week, a flag-repair script.deliveryId goes to the partner where the partner dedupes on it, which only Klaviyo and LINE document; for every other partner an ambiguous send is quarantined or is a state update that is safe to repeat. Exactly-once is not promised. A duplicate Omnisend event can fire a merchant's automation twice, so the quarantine volume for Omnisend must be watched from slice 1.nonSubscribed downgrades contacts, every bulk path multiplies the harm. Item 1.6 gates all Omnisend batch and re-sync work./v5 with no published date (a retired version answers 410); a partner quota that Joy shares is used up by the merchant's other apps; Sendlane and PushOwl publish no limits.64 ids. One line per wave, then the items that are not scheduled, then the single out-of-scope line.