Skip to content This library is free, offered as dana. If it helps your work, donate here →
penguin tree ai
← back to libraryDonate here

This is one of 3,622 free prompts in the penguin tree ai library. Sign up once and the whole library opens, with new prompts by email.

open the library →

API & webhook integration advisor

Recommends a production-ready API and webhook integration architecture for connecting SaaS platforms into an existing tech stack — covering integration pattern selection, authentication strategy, webhook reliability…

download .md

Purpose

Deliver a concrete integration architecture recommendation for connecting a SaaS platform into an existing tech stack — specifying the integration pattern, auth model, webhook reliability design, sync strategy, and failure handling — so the developer builds something resilient the first time instead of discovering the edge cases in production.

When to Use

Invoke this skill when the user asks to:

  • Design an integration between two SaaS platforms or a SaaS platform and an internal system
  • Decide between webhooks, polling, batch sync, or an event stream for a data flow
  • Harden a flaky or unreliable existing integration that drops or duplicates events
  • Choose an authentication strategy (OAuth, API key, JWT, HMAC) for a connection
  • Plan how to connect a martech, CRM, or ecommerce platform into an enterprise stack
  • Handle rate limits, retries, idempotency, or webhook signature verification

Inputs Required

Before starting, collect:

  • Source platform: The system emitting data or events (e.g., Shopify, HubSpot, Salesforce, Stripe)
  • Target system: Where the data needs to land (e.g., data warehouse, CDP, internal DB, another SaaS)
  • Data flow direction: One-way, bidirectional, or fan-out to multiple destinations
  • Data volume & latency needs: Approximate events/records per day and how fresh the data must be (real-time, near-real-time, hourly, daily)
  • Available integration surfaces: Which APIs, webhooks, or connectors each platform exposes
  • Auth constraints: What auth methods each platform supports and any enterprise security requirements
  • Failure tolerance: What happens if an event is delayed, duplicated, or lost (nice-to-have vs. financial/compliance-critical)

If data volume or latency needs are missing, assume near-real-time at moderate volume and state the assumption so the pattern recommendation can be adjusted.

Instructions

Step 1: Select the Integration Pattern

Match the data flow to the right pattern based on latency, volume, and what the platforms actually expose.

  • Webhook (push): Best when the source emits events and near-real-time delivery matters. Requires a public HTTPS endpoint, signature verification, and a queue to absorb bursts.
  • Polling (pull): Use when the source has no webhooks or webhooks are unreliable/incomplete. Poll on a schedule using a cursor or updated_since filter — never full-table scans.
  • Batch/scheduled sync: Best for high-volume, latency-tolerant flows (nightly warehouse loads, catalog syncs). Cheaper and simpler than real-time.
  • Event stream / pub-sub: Use when one source must fan out to multiple consumers, or when replay and ordering guarantees matter. Introduce a broker (Kafka, SNS/SQS, Pub/Sub) between source and consumers.
  • Hybrid (webhook + reconciliation poll): The production-grade default for critical data — webhooks for speed, a periodic reconciliation poll to catch anything the webhooks missed.

State the chosen pattern and one sentence on why the runner-up was rejected.

Step 2: Design the Authentication & Authorization Strategy

Pick the auth model per platform and specify token lifecycle handling.

  • OAuth 2.0: For user-scoped access to platforms like HubSpot, Salesforce, Google. Specify grant type (authorization code for user context, client credentials for server-to-server), token refresh handling, and where tokens are stored (secrets manager, encrypted at rest — never in code).
  • API keys: For simple server-to-server access. Scope keys to least privilege, rotate on a schedule, store in a secrets manager.
  • HMAC signatures: For verifying inbound webhook authenticity — compute the signature over the raw request body using the shared secret and compare in constant time. This is mandatory for any public webhook endpoint.
  • JWT / mTLS: For enterprise stacks requiring stronger identity guarantees.

Call out token expiry handling explicitly — most integration outages are expired-credential failures.

Step 3: Design Webhook Reliability (if webhooks are used)

Webhooks are fire-and-forget from the sender's side. Make the receiver resilient.

  • Verify then acknowledge fast: Validate the HMAC signature, enqueue the payload, and return 200 within the platform's timeout (usually 1–5s). Never do heavy processing before responding — that causes the sender to retry and duplicate.
  • Idempotency: Deduplicate using the event ID or an idempotency key. Store processed IDs so replays and retries don't double-apply.
  • Ordering: Don't assume events arrive in order. Use timestamps or version numbers on the record, and reject stale updates.
  • Retry & dead-letter: Failed processing goes to a retry queue with exponential backoff; permanent failures land in a dead-letter queue for inspection — not silent loss.
  • Replay capability: Persist raw payloads so you can reprocess after a downstream bug fix.

Step 4: Define the Data Sync & Mapping Approach

Specify how records reconcile between systems.

  • Field mapping: Enumerate the source-to-target field map, including type coercions and default values for missing fields.
  • Identity resolution: Define the join key across systems (external ID, email, composite key) and how to handle records with no match.
  • Upsert logic: Use upsert (insert-or-update) keyed on a stable ID rather than blind inserts to avoid duplicates.
  • Deletes & tombstones: Decide whether deletes propagate; many platforms only emit soft-delete flags — handle them explicitly.
  • Conflict resolution (bidirectional only): Pick a strategy — last-write-wins by timestamp, source-of-truth per field, or manual review queue.

Step 5: Handle Rate Limits & Backpressure

Design for the platform's published limits, not optimistic assumptions.

  • Respect published limits: Read the source and target rate limits (per-second, per-minute, daily quotas). Implement a token-bucket or leaky-bucket limiter on the client.
  • Honor 429s: Back off using the Retry-After header when present; exponential backoff with jitter otherwise.
  • Batch where supported: Use bulk endpoints (e.g., Salesforce Bulk API, batch upserts) for high-volume loads instead of per-record calls.
  • Buffer bursts: Put a queue between ingestion and API writes so traffic spikes don't exceed the target's quota.

Step 6: Design Error Handling & Observability

Make failures visible and recoverable.

  • Error taxonomy: Distinguish transient (retry) from permanent (dead-letter + alert) failures. A 500 is retryable; a 422 validation error is not.
  • Structured logging: Log event ID, correlation ID, source, target, and outcome for every event so a single record can be traced end to end.
  • Alerting thresholds: Alert on dead-letter queue depth, sustained error rate above a threshold, and delivery lag exceeding the latency SLA — not on every single failure.
  • Reconciliation job: Schedule a periodic count/checksum comparison between source and target to detect silent drift.

Step 7: Phased Rollout Plan

Ship the integration in de-risked stages.

  • Phase 1 — Read-only shadow: Ingest and log without writing to the target. Validate mapping and volume against reality.
  • Phase 2 — Limited write: Enable writes for a single segment or account; monitor idempotency and error rates.
  • Phase 3 — Full rollout: Enable all traffic with alerting live and the reconciliation job running.
  • Phase 4 — Hardening: Add replay tooling, tune rate limits from observed data, and document runbooks for the on-call team.

Output Format

  • Structure: 7 sections following the diagnostic sequence, with a one-paragraph architecture summary at the top stating the chosen pattern and auth model
  • Length: 1,000–1,400 words
  • Format: Structured document — headed sections with bulleted specifics; field mapping and pattern trade-offs as short tables where they aid scannability
  • Sections: Architecture summary → Integration pattern → Auth strategy → Webhook reliability → Data sync & mapping → Rate limits & backpressure → Error handling & observability → Phased rollout
  • Tone: Solution architect — decisive, specific, trade-off aware; recommends one path and names the rejected alternatives with reasons

Quality Checklist

Before delivering output, verify:

  • [ ] A single integration pattern is recommended, with the runner-up explicitly rejected and why
  • [ ] The auth strategy names the specific method per platform and addresses token/credential lifecycle
  • [ ] If webhooks are used, idempotency, retry/dead-letter, and signature verification are all specified — not just "use webhooks"
  • [ ] Rate limit handling references honoring 429s / Retry-After, not just "add rate limiting"
  • [ ] The data sync section defines the identity/join key and upsert-vs-insert decision
  • [ ] The rollout plan has at least a read-only shadow phase before any writes to the target

Notes

  • The most common production failure is not lost webhooks — it's duplicated events from receivers that process before acknowledging. Prioritize the fast-ack + idempotency pattern.
  • For critical data (payments, orders, compliance), always recommend the hybrid webhook + reconciliation-poll pattern; webhooks alone will eventually drop an event.
  • If the source platform's webhooks don't include the full payload (only an ID), the design must include a follow-up API fetch — account for the extra rate-limit load this creates.
  • Bidirectional integrations are dramatically harder than one-way; if the requirement can be met with one-way sync plus a source of truth, recommend that instead.

How to use this skill

This file is a Claude Code skill in Anthropic's standard format — markdown with YAML frontmatter at the top.

Installing in Claude Code:

  • ~/.claude/skills/<folder-name>/SKILL.md — auto-loads in every session anywhere on your machine (recommended for skills you'll use across multiple projects)
  • <your-project>/.claude/skills/<folder-name>/SKILL.md — auto-loads only when working in that project (recommended for project-specific skills)

Restart Claude Code or open a new session to register.

Using in Claude.ai, ChatGPT, Gemini, or other AI tools:

  1. Drag this file into a new chat, or copy and paste its contents
  2. Send your task or request as your next message — the AI will apply the skill where it fits

For Claude Code skill details: https://docs.anthropic.com/en/docs/claude-code/skills