Consuming webhooks — verify, parse, acknowledge

Summary: a webhook endpoint has three jobs — verify the signature, parse the payload, acknowledge the delivery — and each has a failure mode that looks like silence rather than an error.

A webhook inverts the usual direction: the vendor calls you, at a URL you registered. You don't control the caller, can't replay the call, and their dashboard is often the only record of it.

1. Verify the signature

The URL is public — anyone who finds it can POST to it. So the vendor HMACs the request body with a shared secret and sends the hash in a header; you recompute and compare. A match proves the sender knows the secret and that the body wasn't altered.

• Get the header name from a real delivery, not the docs. Renaming a header breaks live integrations, so vendors keep legacy names for years — GitHub still sends X-Hub-Signature-256; Shortcut (ex-Clubhouse) still sends clubhouse-signature while its docs describe Payload-Signature. Log the whole header dict on delivery one.
• Hash the raw bytes, before JSON parsing — re-serialising changes whitespace, which changes the hash.
• Use hmac.compare_digest, not ==. A normal comparison exits at the first differing byte, so response timing leaks how much of a guessed signature was right.
• Comment the header name, or the next reader will "fix" it to something plausible.

A wrong header name reads as an empty signature, so every delivery 401s — identical to a wrong secret, and a long way from the real cause.

sig = request.headers.get("clubhouse-signature") or request.headers.get("payload-signature", "")

2. Parse the payload as an envelope

One delivery is an envelope — a wrapper around an array of actions, because a single user action often changes several things at once. Each action carries only its own change, so shared context lives elsewhere in the envelope.

• Resolve shared context once, before the loop — not per action.
• Look in order: top-level field → sibling action for the parent entity → references array.
• Fall back explicitly. A .get() returning None into a DB key fails layers later, as a lookup for a record never written.
• Print the payload when it surprises you. Docs describe the common case; the payload is the truth.

# The story ID lives on the 'story' action, not the comment action.
story_action = next(
    (a for a in event.get("actions", []) if a.get("entity_type") == "story"),
    None,
)
event_story_id = story_action.get("id") if story_action else event.get("primary_id")

3. Acknowledge fast, then do the work

Senders wait only a few seconds and treat a timeout or non-2xx as a failed delivery, which they retry — so the same event can arrive more than once (at-least-once delivery). Do real work inline (an LLM call, a DB write, a comment post ≈ 30s) and the sender gives up at five, retries, and the user gets two replies.

• Validate, hand off, return 200 immediately — whatever happens next.
• Make the work idempotent — running it twice has the same effect as running it once. Key it on the event ID (or entity + timestamp) so a duplicate is dropped, not re-run.
• Always-200 deletes your failure signal. The transport now reports success for everything, so error visibility is on you:
  • catch at the entry point and log the full traceback;
  • tell a human where the trigger came from — nobody watches the sender's dashboard;
  • print to stderr with flush=True while bisecting a container, since buffered output is lost when the process dies. Real logger once stable.

try:
    _run_clarify(sc_id, trigger_comment_id)
except Exception as e:
    print(f"[ERROR] _run_clarify failed for {sc_id}: {e}", file=sys.stderr, flush=True)
    traceback.print_exc(file=sys.stderr)
    logger.exception("Error in _run_clarify for %s", sc_id)

Where this came up

Building a Shortcut bot, one bug per job: the signature was read from X-Shortcut-Hmac-SHA256 (a name invented from the branding, so every delivery 401'd); the story ID was read from the story-comment action, which doesn't carry it; and a crashing handler surfaced nothing, because the endpoint returns 200 unconditionally so Shortcut won't double-process.