Webhook Deduplication

In this guide we cover:

• Standard duplication undertaken with webhooks
• Additional Deduplication that can be switched on
• Deduplication scenario
• Opting In
• Points to note

Standard Deduplication

The standard webhook solution employs a form of deduplication, whereby we attempt to deduplicate “burst events” over a relatively short time period. E.g. If we observe multiple update events on a variant when the countOnHand changes, we will attempt to send only the last relevant event, as opposed to sending every countOnHand change.

The standard solution works across a relatively small timebox as we need to balance the desire to deduplicate events, with the timely delivery of those events.

This part of the solution will remain intact and is not changing, the opt in feature (described below) would operate in addition to this.

Additional Deduplication

While the standard approach to deduplication removes some unnecessary events, there are further use-cases where redundant events manifest over longer periods of time, in particular where webhook event payloads do not change.

This new feature seeks to address that use-case, and is best illustrated through the use of an example.

Scenario

In the following example, we have 2 webhooks set up on Marketplacer feeding 2 separate downstream customer systems:

• The Advert webhook feeds the Product DB, ensuring slower moving Product data for newly created and updated products flows in near real time.
• The Variant webhook feeds the Inventory Management system. Inventory position changes would be considered fast moving data, also provided in near real time.

The Opportunity

In this current set up, updates on the Variant (e.g. countOnHand / inventory position changes), also result in a corresponding update event on the Advert - which is expected behaviour. This can lead to scenarios where:

• We legitimately require the Variant Update webhooks
• We don’t require redundant Advert Update webhooks triggered by fast-moving Variant updates

The opt-in deduplication feature seeks to deduplicate webhook events where:

• The data payload for the current event is identical to the last event for that object, and
• The event type is the same (e.g. Update)

So for example if the Advert webhook employed above was defined as follows:

query ($id: ID!) {
  node(id: $id) {
    ... on Advert {
      __typename
      id
      legacyId
    title
    fullTitle
    vetted
    seller {
      businessName
    }
    variants {
      nodes {
        id
        label
      }
      }
    }
  }
}

This would result in the following data payload when (for example) an update occurs:

{
  "id": "V2ViaG9va0V2ZW50LTIwNw==",
  "event_name": "update",
  "payload": {
    "data": {
      "node": {
        "id": "QWR2ZXJ0LTEwMDAwMjYxNA==",
        "title": "T-Shirt Red",
        "seller": {
          "businessName": "Cell Phones 4U"
        },
        "vetted": false,
        "legacyId": 100002614,
        "variants": {
          "nodes": [
            {
              "id": "VmFyaWFudC0zNjcx",
              "label": "Large / Green"
            }
          ]
        },
        "fullTitle": "Bucket Placer T-Shirt Red",
        "__typename": "Advert"
      }
    }
  }
}

If a subsequent Advert Update event is triggered, but the data payload does not result in any changes to the data, then this new solution would skip sending this event.

An example taken from the Webhook audit log shows what this may look like:

Opting in

You can enable this deduplication feature on some or all of your webhooks by simply checking Allow Skip?:

Points to note

• You can Requeue Skipped events from the portal, which will attempt to resend the webhook irrespective of whether the data payload is considered a duplicate
• Contents of the Data Payload that change along with fast moving updates will render the data payload unique and will not result in this deduplication routine running. Examples of such fields would be:
  • If you chose to also include the value of countOnHand in the Advert payload
  • If you chose to include the updateAt date in the Advert payload
• We have focused on a scenario involving Adverts and Variants, but this solution would work for all webhook object types
• If deployed, this feature could be enabled and disabled by the Operator