Joy Loyalty · Integrations
What is breaking, why, and the work to make every integration deliver, recover and tell the merchant.
The problem in numbers · prod logs 02/11 → 19/11
authorization header. Recharge addresses and Apple Wallet auth headers leak toookCounts are log lines, not unique customers. Shop ids and emails stay out of this deck.
One cause behind all of it
There is no integration module. Each app is hand-wired across ~30 files, on a path that cannot retry, with no delivery id, no dead-letter queue, no health state and no success metric.
Compute, Pub/Sub and Cloud Tasks scale well past 5 to 8 more partners. What is missing is everything that makes a delivery safe.
Why rebuild instead of patching
The core change, on one delivery
Today
retry:falseundefined; dispatcher returns 'ok'After
deliveryId, then acksSame pattern for auth: a 401 becomes auth_broken (calls stop, tokens kept, merchant told), not weeks of retries.
1 · Integrations in use · prod Firestore, read-only count, 20/11
What the count itself shows
integration doc vs shop flag
integration doc vs shop flag
connections on shops that uninstalled Joy
connections on shops that uninstalled Joy, matching the dead-refresh calls
Today
Read from the code at HEAD 2ff18f82dc.
2.1 · How an integration is wired today
App id with the loyalty-program constants; events, message and job types elsewhere. Ids use four casings.
const/programs/integrationPrograms.js
Repository connect/disconnect, controller paths, fan-out lists, the Cloud Task handler map, and admin cards, routes, status map.
subscribeBackgroundHandling.js: 212 cases, 9,933 lines
Shop statusConnected* (19 names), doc isConnected, admin fallback map, AI tool map. They disagree, as measured.
Connect callbacks, disconnect, repo side effects, Klaviyo auto-disconnect (revokes!), Intercom auth failure, Drip bulk failure, generic update.
2.2 · Outbound path today, and where work is lost
triggerIntegrationAppServiceretry: false. One dispatcher with 212 casesmaxAttempts: 1, no rateLimitsInbound is no better: four review sources are unsigned, two verifiers fail open, Recharge is unauthenticated, and an HMAC failure on a Shopify webhook returns 200 with no log.
3.2 · Infrastructure per integration type
| Type | Apps | Runs on today | On 401 / 429 / 5xx today |
|---|---|---|---|
| Email / SMS marketing (out) | Klaviyo, Omnisend, Mailchimp, Attentive, Drip, Sendlane, PushOwl | Pub/Sub Light/Small (retry:false) → service; Klaviyo re-enqueue on the shared queue; Klaviyo bulk collectors | Throttle gate, 3 tries, drop; only Klaviyo's private key disconnects on 401 |
| Helpdesk / messaging | Gorgias, Intercom, LINE, Smax | Same Pub/Sub path; Gorgias limiter in RAM | Gorgias refresh failure → TypeError loop; LINE retries without an idempotency key |
| Reviews (in) | Judge.me, Yotpo, Loox, Stamped, Fera, Review Rocket, Air Reviews, Klaviyo Reviews | HTTP webhooks on apiHookV2, inline; generic dedupe | 4 unsigned, 2 fail-open |
| Subscription / wishlist | Recharge, Joy Subscription, Joy Wishlist | Inbound HTTP + outbound calls | Recharge unauthenticated, tenant from the query, PII in logs |
| Social | Instagram, Facebook Page | Signed Meta webhooks → Pub/Sub; token refreshed on demand, no cron | Revoked token retried on every use |
| Wallet | Apple, Google Wallet | HTTP + APNs / Google API | Auth headers and push tokens still logged |
| Joy email | Mailgun custom domain, merchant SMTP | sentEmailNotification topic, no retry | The only full breaker in Joy, but custom domain tried on every send (~18k/day) |
| Shopify (integration paths) | Order webhooks, Admin API, Flow | apiHookV2 + Cloud Tasks for orders | Enqueue failure answers 200; ~10.7k throttled Admin calls/day; Flow 200-on-failure |
| Joy-provided APIs | REST v2, outbound webhooks, headless, MCP | HTTP functions; webhooks via webhookSend, no retry | Non-JSON 2xx counted as failure; merchant 5xx never retried |
3.3 · Is today's infrastructure enough?
Integration topics are retry:false; the shared queue allows one attempt
3 apps share integrationRateLimits; 30 s per-instance cache
Klaviyo unique_id falls back to a random id; no task has a name
Only the ordered topic has one, via a manual script not in CI
Only SMTP has one. 61,240 Klaviyo refresh-failure lines
Apps log failures as free text and never log successes
4 unsigned sources, 2 fail-open, 1 unauthenticated
Tokens in message payloads and logs; one constant key in source
Evidence
Cloud Logging, 02/11 → 19/11 (the bucket keeps 18 days), read-only, paginated to exhaustion. High-volume fingerprints counted exactly.
4.2 · Root-cause rollup · 336,363 counted lines
Not in the shares (different sample windows): false webhook failures 10,491 · custom-domain fallback ~18k/day · Shopify Admin throttling ~10.7k/day · order webhooks not queued 344.
4.1 · Findings by integration
contacts + events dropped after 429. Retry sleeps in the handler, then returns
profile syncs dropped; deferrals count toward 3 tries. 98% from the 13/11 burst
dead grants refreshed for weeks; the invalid_grant branch is unreachable
invalid_client since 05/11 on 8 accounts, then a TypeError loop
401/403 in 10 days, every line with the merchant's auth header
revoked session retried on every use; no cron, no reconnect prompt
not queued, 200 returned. Whole ctx.state put in the task
throttled calls inline in webhooks; customer reads return empty, tags dropped
false failures: 2xx with a non-JSON body becomes 500
custom domain tried for shops with no verified domain
Target
Built once, proven on one app first (Omnisend events for delivery, Klaviyo for health), then each app moves behind a flag.
5.1 · Target architecture
5.1b · Every partner is different · Joy sends out
The differences live in each app's adapter; the delivery worker only asks it "when may I retry?" and "does the partner dedupe?".
| Partner | Auth | Limit | When busy (429) | Dedupe | Specific |
|---|---|---|---|---|---|
| Klaviyo | OAuth (or private key) | Per account + app, by endpoint tier (events 350/s, bulk 10/s) | Retry-After header | Yes: unique_id | Token refresh max 10/min; bulk cannot clear a property |
| Omnisend | API key / OAuth | 400/min per brand, shared with other apps | retryAfter in body (some endpoints) | No dedupe for automation events | Moved to a date-versioned API; Joy still on /v5 |
| Mailchimp | OAuth | 10 concurrent connections (not a rate) | None | Upsert by email hash, safe to repeat | Batches are unordered |
| Attentive | OAuth | Subscribe 10/s | x-ratelimit-* headers | Undocumented | Re-subscribing texts the customer |
| Gorgias | OAuth | 80 requests / 20 s | — | — | Joy limits itself to 35/20 s in memory; invalid_client since 05/11 |
| Meta OAuth | — | — | — | Token lives 60 days; refresh before expiry | |
| LINE | Channel token | Monthly quota | No header | Yes: X-Line-Retry-Key (24 h) | A 429 may mean the monthly quota is gone |
| Postscript (new) | Partner key + shop key | 15/s per partner–shop pair | Error code 3002, no header | external_id dedupe undocumented | Max 50 properties; never subscribe customers |
5.1c · Every partner is different · partners send to Joy
The verifier has one contract, but how a signature is checked is specific to each provider.
| Source | How to tell real from fake | When Joy answers an error | Dedupe / note |
|---|---|---|---|
| Shopify webhook | HMAC on the raw body | 5 s timeout; 8 retries over 4 h, then the subscription is deleted | X-Shopify-Webhook-Id (Joy uses Event-Id) |
| Shopify Flow action | HMAC | Retried up to 36 h, incl. timeouts and 429 | action_run_id |
| Judge.me | JUDGEME-HMAC-SHA256, Joy's OAuth app secret | — | — |
| Loox | HMAC only if the merchant sets a key (Convert/Unlimited) | — | — |
| Fera | JWT in the URL, signed with the store secret Joy holds | — | Verifiable today |
| Yotpo | Unsigned → secret token in the URL | At least 10 retries | WebhookMessageId |
| Stamped | Secret in a header, no body signature | — | — |
| Recharge | X-Recharge-Hmac-Sha256 = SHA-256(secret + body); Joy lacks the secret | Retries, then deletes the webhook | — |
| Loop (new) | X-Loop-Signature, merchant's secret | 10 s timeout; 5 retries; at-least-once | Topic return, trigger return.closed; exchange must be derived |
Source: each partner's official API docs, checked on 20/11 (verify-partner-api-A/B.md).
5.3 · Seven design principles
A new app means writing that app's module, not editing 30 files.
Not chosen: app settings stored in the database (still needs code, and is hard to test).
Each send has its own id. Partners that dedupe (Klaviyo, LINE) receive it once; for the others, a send with an unknown result is held for review.
Not chosen: turning on retry for the shared queue (it would re-run hundreds of unrelated jobs).
App by app, starting with Klaviyo.
Not chosen: switching everything at once while 7 other places still write it.
The sender loads the token when it runs, so old work never uses a newer install's token.
Not chosen: moving every token to Secret Manager now (too big, blocks other work).
The merchant issues a key per partner, and it opens only the APIs that partner needs.
Not chosen: sharing the shop's key (full access).
Each app has a switch; if something breaks, turn it off and the old path runs again.
Not chosen: one switch for everything, changed in one go.
No token deletion, no automatic disconnect; once the merchant reconnects, it carries on.
Not chosen: disconnecting and revoking access, as Klaviyo does today.
Send reliably, avoid duplicates, know which app is broken, and keep a way back for every change.
The work
5 days, before Black Friday
Klaviyo, Omnisend, Gorgias…
send queue, connection health
Flow, Tapcart, Loop, Postscript
and removing dead code
Each item's detail (what, where in the code, when it is done) is in roadmap.md.
Group 1
20/11 → 25/11Group 2
Waves 1 – 3How it is fixed · 1 / 2 · sending data and connections
Joy retries a few times within seconds, then gives up. 205,000 times in 18 days.
Every send goes through one queue. A busy partner means "send later" by that partner's own rule, never "drop". Budgets sit below the published limit because quota is shared with the merchant's other apps.
No data lost; unclear results held for review
One shop's mass tier demotion caused tens of thousands of calls, one per customer.
Group them into one bulk sync job that Klaviyo processes once.
No more floods like 13/11
Klaviyo, Gorgias, Instagram, Mailchimp keep being called for weeks without access; the merchant still sees "Connected".
On lost access, stop calling, keep the token, show the merchant a banner. On reconnect, sync again automatically.
The merchant knows and can fix it
Every customer page view asks Klaviyo/Omnisend, even when the shop never connected them.
Ask only the apps the shop has connected.
80% fewer Klaviyo errors in the logs
Since 05/11 the token refresh fails and triggers another error, ~8,900 lines a day.
A failed refresh marks the connection as lost; separately, check Joy's own Gorgias OAuth account.
Loop ends; the 8 failing accounts are told
One failing shop stops the whole cron; jobs stay "syncing" forever.
Each shop runs on its own; a job that fails too often is marked failed.
Other shops are not dragged down
How it is fixed · 2 / 2 · Shopify, webhooks, email, security
Large orders could not enter the queue, yet Joy told Shopify "received". 344 orders.
Save the order first, then answer Shopify. The queue holds only the order id, so it is never too large. If saving fails, answer an error so Shopify resends.
No order lost
When Shopify throttles, Joy treats it as "no data" and skips tagging. ~10,700 times a day.
Throttled calls are queued and retried; large tag runs use bulk operations.
No missing tags or customer data
A merchant server answering "ok" counts as an error, while real errors are never resent.
A success answer is a success. Real errors are resent later.
10,491 false errors gone
Logs contain keys, headers, customer addresses and recipient emails.
Logs keep only the error type, code and ids.
No new log lines exposing secrets
Four review apps send unsigned webhooks; Recharge is unauthenticated. Anyone with the URL can send fake reviews.
One method per provider: Fera already signs (check now); Recharge needs the Recharge app secret; Yotpo cannot sign, so a secret in the URL. A transition period avoids blocking real data.
Fake reviews and orders blocked
Flow reports success on failure; email tries a custom domain the shop does not have (~18,000 times a day); Joy Subscription ignores Shopify resends.
Flow reports errors clearly; email checks for a verified domain first; Joy Subscription marks "processed" only after saving.
Errors become visible; 95% less email noise
Hidden issues · from partner docs, not yet in logs
Found by checking each fix against the partner's official API docs. Each card: what can go wrong → what to do.
If Joy's handler answers non-2xx, Recharge retries and then deletes the webhook. Joy does not store the secret needed to verify Recharge.
Check the status on staging; choose the secret source (ask shops, or a URL secret) when building the verifier.
Each bulk sync calls subscribe for people already subscribed; Attentive sends them an SMS.
Subscribe only consented, not-yet-subscribed customers; cap 10 calls/s.
/v5Joy still calls /v5; Omnisend moved to a date-versioned API (2026-03-15), no retirement date yet.
Build new adapters on the new API; ask Omnisend for the /v5 date.
A resent event can fire a merchant automation twice.
Ambiguous sends are quarantined, never blindly resent.
New list members are added as "subscribed" with no consent check.
Review together with the Attentive consent fix.
Shopify documents X-Shopify-Webhook-Id; Joy dedupes on X-Shopify-Event-Id across topics.
Switch the key when reworking order webhooks.
Joy refreshes on every 401 with no lock; a 429 there can be misread as a dead grant.
Serialise refreshes per shop; a token-endpoint 429 is transient, never auth_broken.
Stored batches that nothing processes.
Winn confirms; drain them or remove the code.
Loop has no "customer chose exchange" field; Postscript does not document dedupe on external_id.
Prove on real payloads / a test shop in the spike first.
Group 3
proven on one app firstEvery app sends through one place. A busy partner gets the send later, by its own rules; nothing is silently dropped. Omnisend first.
Know which app is fine, flaky or disconnected. When disconnected, stop calling and keep the token. Klaviyo first.
A banner such as "Klaviyo is disconnected, birthday emails are paused", with Reconnect and Re-sync buttons.
Accept only real data, from the right partner, for the right shop.
Large actions run as one job, not tens of thousands of calls.
See a shop's failed sends and press resend.
Each app declared in one place, with a test that catches a forgotten edit.
Success rate per app, with an alert when it drops.
Group 4
New appsOkendo, REVIEWS.io, Junip, Growave. Quick to build; Rivo does it this way too.
5 of 9 competitors have it. Fix the exposed key first.
Listing first. Joy's edge: no competitor gives points when a customer picks an exchange over a refund.
Top pick, 6 of 9 competitors. The send queue must be done first so no SMS is sent twice.
Skio, Loop Subscriptions, Stay AI: six other loyalty apps are listed; Joy is not.
Group 5
Partner API & clean-upThe merchant creates one key per partner. A key opens only what is needed: customers, tier, rewards, add/remove points. Each partner has its own call limit, and the key can be revoked at any time. Loop, Tapcart and Okendo will use it.
Decide by 25/11 which plans may use the partner API
7 · Order and timing
Tết break 04/02 → 10/02. Big changes start only from the week of 04/01.
7.2 · Turning changes on safely
| Change | Must be able to see | When to roll back | How to roll back |
|---|---|---|---|
| First small fixes | Alerts for dropped data and Klaviyo errors | A new kind of error appears | Redeploy the old version, under 30 minutes |
| Send queue, app by app | Successful / retried / failed sends per app; any duplicates | Success lower than the old path, or a duplicate | Switch that app off, under 15 minutes |
| "Disconnected" status | How many shops are flagged by mistake | False flags above the limit | Switch off, run the script that restores the old status |
| Order webhooks answer real errors | One week of observe-only | More errors than during observation | Redeploy the old version, re-register webhooks |
| Signatures required | Which shops still send unsigned | Real data blocked | Turn on accept-both mode again |
| SMTP re-encryption, Tapcart key change | Decode errors, email send errors, shop list | Errors spike | Old format stays readable; change one shop at a time |
Every change merchants can see ships with a help article, a support reply template and one line of explanation in the app.
8 · Decisions needed · only the ones that block work
Hire, borrow a developer for 8–10 weeks, or run the reliability-first fallback. Winn · before 02/12 · blocks Flow, Loop, Postscript, F1
Partners on Free? Shared quota? Custom actions Ultimate-only? Winn + product · 25/11 · blocks F1, Loop, Tapcart
Still valid? Test: refresh one healthy account. What changed on 05/11 01:46 UTC? Winn · 25/11
False-positive limit, ever auto-disconnect?, is degraded visible? Winn + product + support · before 04/01
Avoid duplicates, avoid loss, or reconcile by hand; maximum replay age per partner. Omnisend events are quarantined by default meanwhile. Wave 2 · blocks Postscript
9 · Risks
One week observe-only first; tokens are never deleted, so a false flag breaks nothing
Each send has its own id; sends with an unknown result are held for review
If true, bulk sync multiplies the damage → no bulk sync until the test result is in
Breaks the old block → change shop by shop, after the new block is live
Before sending, always check the customer may still be sent
Sample only, with a budget, in a separate bucket
Tests fail for the wrong reason → prepare queues/indexes on staging first
Errors that were silent now show → compare with that in mind
No developer; slow partner approvals; slow Chatty update
Cut scope, never skip a safety step
Next 5 days
Source: roadmap.md v7 and integration-usage-counts.md, plans/261119-2330-integrations-roadmap.