1. Data Flows
Understand the flows of data for a typical connected integration.
15 minute read
What you’ll learn
In this section of the Connected Integration Blueprint (CIB) you’ll learn about:
- The core data entities of the Marketplacer platform
- The core relationships between those entities
- The core flows of data in and out of a Connected Integration
- (Between Markertplacer and your eCommerce solution)
Core Entities
Marketplacer is a large product, comprised of many entities, so describe them all here in detail would be counter to the nature of this playbook. Instead, we will focus on the core (or primary) entities that will be of interest to you when integrating into Marketplacer.
Further Reading
If you’d like to explore all the entities that are expose via the Operator API, then we’d suggest you take a look at GraphQL Voyager.The table below describes the core entities we’ll be discussing in the rest of the CIB.
| Entity | Alias | Purpose | Context |
|---|---|---|---|
| Advert | Product | Primary container for Products that a Seller wishes to sell on the marketplace. These products would be referred to as 3rd party products when sold along side an operators own 1st party products. | Products |
| Variant | Describes how the Advert can vary, e.g. different sizes or colors. An Advert must always have at least 1 variant. It is at the variant level where attributes such as: SKU, Bar code, Inventory are located, and ultimately it is the variant that is purchased by the customer. | Products | |
| Order | An Order is the “marketplace-level” object that acts as the container for a customer’s entire purchase. An Order in Marketplacer will always have at least 1 Invoice attached to it, (1 invoice per seller) | Orders | |
| Invoice | Order | An Invoice is the part of the overall customer order that an individual Seller has to fulfill. An Order will always have at least 1 Invoice (1 Invoice per Seller participating in the Order). From a Sellers perspective the Invoice is the Order. | Orders |
| Line Item | Line Items represent the products purchased by the customer. Line Items have a direct line of sight back to the Variant. I.e. when a variant is purchased it is then represented as a Line Item. Line Items are attached to both the Order and the Invoice. | Orders | |
| Refund Request | A Refund Request models the stages that both Cancellation (pre-dispatch) and Refund (post-dispatch) requests go through. A successful Refund Request will result in an Invoice Amendment. | Orders | |
| Invoice Amendment | Refund | Represents an adjustment to an Invoice. Usually generated after a successful Cancellation or Refund Request. | Orders |
| Shipment | Represents the dispatch of 1 or more Line Items, Sellers are typically responsible for the creation of shipments. | Shipments | |
| Shipped Item | You can choose to split shipments, so either individual line items on an Invoice, or even specific quantities of the same line item. This is represented by a Shipped Item | Shipments | |
| Seller | Retailers | Sellers are 1 of the primary participants on the marketplace and are responsible for the listing of (3rd party) products for sale. | Sellers |
| Remittance | The marketplace operator is responsible for taking payment for orders, remembering that orders can be comprised of products from multiple sellers. This is referred to as the Merchant of Record. At some point however the Sellers participating in marketplacer orders will need to be paid (by the marketplace operator), this payment is referred to as a Remittance. | Sellers | |
| Taxon | Category | Represents where the product sits in the categorization hierarchy, e.g. Computers -> Storage -> SSH Hard Drives. In this example we have 3 Taxons, with Parent / Child relationships between each. | Categorization |
| Option Type | Attribute | Option Types refer to the the way a Variant can be defined. So common Option Types would be color and Size. Option Types will then have 1 or move Option Values, which contain the range of values this it is possible for that Option Type to have. | Categorization |
| Option Value | Attribute Value | The values that an Option Type can have. In the example where we have an Option Type of Color, possible Option Values would be: Red, Green, Blue etc. | Categorization |
| Prototype | Attribute Profile | Prototypes are the glue between a Taxon and the Option Types that Taxon has. So we assign Option Types to a Prototype, and then assign a Prototype to the Taxon. This means we can assign that same defined Prototype to any number of other Taxons. | Categorization |
Entity Relationships
We’ll start by looking at each of the entity relationships grouped by context, this is easier to digest at first, and is arguably the most useful view anyway!
Products
- An Advert must have 1 or more Variants
- A Variant must have 1 Advert
Orders
- An Order must have 1 or more Invoices
- An Order must have 1 or more Line Items
- An Invoice must have 1 Order
- An Invoice must have 1 or more Line Items
- [Optional] An Invoice can have 1 or more Invoice Amendments
- [Optional] An Invoice can have 1 or more Refund Requests
- A Line Item must have 1 Order
- A Line Item must have 1 Invoice
- A Refund Request must have 1 Invoice
- An Invoice Amendment must have 1 Invoice
Shipments
- A Shipment must have 1 or more Shipped Items
- A Shipped Item must have 1 Shipment
Sellers
- [Optional] A Seller can have 1 or more Remittances
- A Remittance must have 1 Seller
Categorization
- [Optional] A Taxon can have 1 Prototype
- [Optional] A Prototype can have many Taxons
- [Optional] A Prototype can have many Option Types
- [Optional] An Option Type can have many Prototypes
- [Optional] An Option Type can have many Option Values
- [Optional] An Option Value can have 1 Option Type
Extended View
Note
The extended view below aims to highlight the primary relationships between contexts, there may be instances however where not every relationship is shown. This has been done deliberately to maximize the readability of the domain. Again, please refer GraphQL Voyager for the full range of relationships and entities.
We have just listed the cross-context relationships below:
- [Optional] A Seller can have 1 or more Adverts
- An Advert must have 1 Seller
- [Optional] An Invoice can have 1 Remittance
- A Remittance must have have 1 Invoice
- [Optional] An Invoice can have 1 Shipment
- A Shipment must have 1 Invoice
- A Line Item must have 1 Variant
- [Optional] A Variant can have 1 Line Item
- [Optional] A Line Item can have have 1 Shipped Item
- A Shipped Item must have 1 Line Item
- [Optional] A Variant can have many Option Values
- [Optional] An Option Value can have many Variants
- [Optional] An Advert can have 1 Taxon
- [Optional] A Taxon can have many Adverts
We will be referring back to this model throughout the rest of the CIB.
Data Flows
Before talking about the specific individual Data Flows, it’s worth revisiting the high level Product/Order flow of a typical marketplace:
- Sellers will list products on the marketplace
- Seller products (aka 3rd Party Products) will need to flow through to the eCommerce platform
- Customers will browse products on the front end (3rd party and possibly 1st party)
- Customers will place an order on the eCommerce platform (payment is taken here)
- Orders (for 3rd party products only) are passed through to Marketplacer
- Sellers will pick up their orders and process
Coverage
We’ve only discussed the (very!) high level data flows for Products and Orders so far, we will of course cover these flows and others (e.g. Shipments, Refunds etc.) in more detail in the sections that follow. However, while every effort has been made to include a comprehensive list of flows that would be applicable to most scenarios, you may of course have additional requirements that are not listed.Connected Integration Context
For the purposes of this section, and this blueprint as a whole, we will be focused in on the following data flow context:
Therefore in this section we are only going to talk about the movement of data between Marketplacer (MP) and the Connected Integration (CI) and vice versa. In later sections we will however discuss:
- Which APIs and Webhooks you can use to move this data (Section 2 - Integration Points)
- Approaches to how you could compose your integration (Section 3 - Architectural Patterns)
It’s also worth clarifying the following terms when we talk about the flow of data:
- Information Provider
- Information Consumer
Information Provider
The Information Provider (referred to as just the “Provider” from now on) is the source of the data to be moved. E.g. If a product is created by a Seller in Marketplacer and that Product needs to propagate to the Connected Integration, then:
- Marketplacer (MP) is the Provider
- The Connected Integration (CI) is the consumer
Information Consumer
The Information Consumer (referred to as just the consumer from now on) is the target of the data to be moved. We can refer to the example given above to understand the role of the consumer.
Our context can be summarized as follows:
An expanded view showing the high-level flow of the entity contexts is shown below:
Where do we start
We’ve listed the data flows below in a particular sequence as this typically how we’d recommend marketplace operators build their integrations. You may of course want to focus on other flows first, however we’ve found the suggested sequence below seems to work well in most situations.Product Flows
Let’s remind ourselves of the product entities:
At a simple level Advert and Variant data will need to flow from Marketplacer (MP) to the Connected Integration (CI). Less common, although certainly valid for some use-cases, Advert and Variant data may flow back from the Connected Integration to Marketplacer. Let’s explore some specific product use-cases in more detail:
| Provider | Consumer | Use Case | Description |
|---|---|---|---|
| MP | CI | Product Created | Whenever a seller creates new products in Marketplacer these will need to flow through to the Connected Integration. |
| MP | CI | Product Updated (Ad Hoc) | Existing products will be updated for various reasons, these updates will need to flow through to the Connected Integration |
| MP | CI | Product Deleted | Existing products may be deleted by the Seller, and will therefore no longer be available for sale, this will need to flow to the Connected Integration |
| MP | CI | Product Off-lined | Sellers may choose to “offline” a product if they no longer want to take orders for it, this is different from deletion as they may want to bring the product back online at some point. |
| MP | CI | Inventory Updated | Inventory updates occur at the variant level (this is where we hold Inventory Position). This is a unique product update use-case in that the cadence of updates can be high, there are also potentially larger implications for operator and seller if this data is not accurately reflected across domains. |
| MP | CI | Pricing Updated | This is essentially just another product update, but has been called out specifically given the potential for regular of updates, as well as he criticality of that data. |
| MP | CI | Advert Vetted | Operators can choose to enable Advert Vetting, meaning that products are not automatically placed in an online state and made available for sale. If an Operator chooses to vet a product using the Marketplacer Operator Portal, that update should propagate to the CI. |
| CI | MP | Advert Vetted | Same background context as above, but Operators can choose to vet products using the Marketplacer API, in which case this data is flowing form the CI to Marketplacer. |
| CI | MP | Product Enriched | There are some use-cases where operators have specific requirements around how products are listed, in some instances this means that operators enrich seller product listings using the Marktplacer API to adhere to their requirements. |
Order Flows
Let’s remind ourselves of the order entities:
We have chosen to include Refunds as part of the Order context, and as such we’ll cover the data flows here. It’s worth noting 2 points however:
- Refunds are usually tackled later in the development cycle
- You could arguably split out Refunds into a separate context - feel free to do so if this fits your use-case.
| Provider | Consumer | Use Case | Description |
|---|---|---|---|
| CI | MP | Order Created | One of the core flows of the entire integration, the movement of order data through to MP, note that Invoices are created at essentially the same time as the order, therefore we do not have a separate Invoice Created flow. |
| CI | MP | Order Updated | Order updates from the CI to MP are rare, indeed the only reason it hs been captured here is to call out that it almost never happens. You may want to refer to some of the refund flows for use-cases around cancelling and refunding orders. |
| CI | MP | Refund Request Created | Refund Requests cover both order cancellations (pre-dispatch) and refunds (post dispatch). Requests can be raised via the CI (using the Operator API), and propagate to MP. |
| CI | MP | Refund Request Updated | Refunds (and Cancellations), go can go through 1 of 2 workflows depending on your Marketplacer configuration. Either way, in both cases updates to the Refund Request can flow from the CI to MP. Requests can be raised via the CI (using the Operator API), and propagate to MP. |
| CI | MP | Refund Request Approved | Finalizing the refunds workflow can result in the approval of the refund request. Requests can be raised via the CI (using the Operator API), and propagate to MP. |
| CI | MP | Refund Request Denied | Finalizing the refunds workflow can result in the denial of the refund request. Requests can be raised via the CI (using the Operator API), and propagate to MP. |
| MP | CI | Refund Request Created | Refund Requests cover both order cancellations (pre-dispatch) and refunds (post dispatch). Requests can be raised using the Seller and Operator Portals in MP, if this occurs then this data can flow back from MP to the CI. |
| MP | CI | Refund Request Updated | Refunds (and Cancellations), go can go through 1 of 2 workflows depending on your Marketplacer configuration. Either way, in both cases updates to the Refund Request can flow from the CI to MP. Requests can be raised using the Seller and Operator Portals in MP, if this occurs then this data can flow back from MP to the CI. |
| MP | CI | Refund Request Approved | Finalizing the refunds workflow can result in the approval of the refund request. Approvals can be raised using the Operator Portal in MP, if this occurs then this data can flow back from MP to the CI. |
| MP | CI | Refund Request Denied | Finalizing the refunds workflow can result in the denial of the refund request. Denials can be raised using the Operator Portal in MP, if this occurs then this data can flow back from MP to the CI. |
| MP | CI | Invoice Updated | Along side the creation of the order, the update of the Invoice is one of the key data flows found in the integration. Invoices can be updated for a number reasons, including but not limited to: when a refund is approved, when a shipment is created and any other event that would cause a state transition. |
Shipment Flows
Let’s remind ourselves of the shipment entities, (note we have included the relationship to the Invoice as it’s such a key relationship in the context of Shipments):
| Provider | Consumer | Use Case | Description |
|---|---|---|---|
| MP | CI | Shipment Created | When a Seller dispatches all or part of an order a Shipment will be created in MP, this data will need to flow to the CI |
| MP | CI | Shipment Updated | Shipments do have a status, which can be updated throughout the lifecycle of the Shipment, these update events should flow through to the CI. |
Seller Flows
Let’s remind ourselves of the seller entities:
| Provider | Consumer | Use Case | Description |
|---|---|---|---|
| CI | MP | Seller Created | Sellers can be created via the Operator API, there may be scenarios where this is employed (e.g. if you have a Seller Sign Up page). |
| CI | MP | Seller Updated | Updates to existing Sellers may propagate from the CI to MP, e.g. you may want to auto offline a seller if they have breached you marketplaces terms and conditions |
| MP | CI | Seller Created | Where new Sellers are created an on-boarded using the MP Operator Portal, you may want to propagate this date through to your CI. |
| MP | CI | Seller Updated | Where existing Sellers are updated using the MP Operator Portal, you may want to propagate this date through to your CI. |
| CI | MP | Remittances Updated | Before Sellers can be paid, the operator needs to approve their release. You can automate this release process via API in which case the flow of remittance release data is from the CI to MP. |
| MP | CI | Remittances Created | Remittances are created when certain conditions are met by a seller for their invoices, when these conditions are met, the remittance is created, but not yet released, in this case you may still want the unreleased remittance data flow from MP to the CI. |
| MP | CI | Remittances Updated | The remittance release process can originate in MP, in which case you may require that data to flow from MP to the CI. |
Categorization Flows
Let’s remind ourselves of the categorization entities:
Note
While categorization data can flow in both directions (MP to CI and CI to MP) the flow of this data is less commonly implemented than the other flows (if at all).| Provider | Consumer | Use Case | Description |
|---|---|---|---|
| MP | CI | Taxon Created | New Taxons (categories) created in MP may need to flow through to the CI |
| MP | CI | Taxons Updated | Existing Taxons (categories) updated in MP may need to flow through to the CI |
| MP | CI | Prototype Created | New Prototypes created in MP may need to flow through to the CI |
| MP | CI | Prototype Updated | Existing Prototypes updated in MP may need to flow through to the CI |
| MP | CI | Option Type Created | New Option Types created in MP may need to flow through to the CI |
| MP | CI | Option Type Updated | Existing Option Types updated in MP may need to flow through to the CI |
| MP | CI | Option Value Created | New Option Values created in MP may need to flow through to the CI |
| MP | CI | Option Value Updated | Existing Option Values updated in MP may need to flow through to the CI |
| CI | MP | Taxon Created | New Categories created in the eCommerce platform may need to be created in MP |
| CI | MP | Taxons Updated | Existing Categories updated in the eCommerce platform may need to be updated in MP |
| CI | MP | Prototype Created | New Prototypes created in the eCommerce platform may need to be created in MP |
| CI | MP | Prototype Updated | Existing Prototypes updated in the eCommerce platform may need to be created in MP |
| CI | MP | Option Type Created | New Option Types created in the eCommerce platform may need to be created in MP |
| CI | MP | Option Type Updated | Existing Option Types updated in the eCommerce platform may need to be created in MP |
| CI | MP | Option Value Created | New Option Values created in the eCommerce platform may need to be created in MP |
| CI | MP | Option Value Updated | Existing Option Values updated in the eCommerce platform may need to be created in MP |
What’s Next
Having identified the most common data flows for a Connected Integration, in the next section we look at what methods you can use to move that data.