Keeping your catalog fresh

A catalog is never loaded once and left. Offers keep changing at Groupon, so your copy needs a schedule:

What changes between syncs

• Prices: an option's retail price moves, and a promotion starts or ends (option.promotion becomes set or null).
• Availability: an offer sells out (sold_out) or an option is switched off (active: false), and it can come back.
• Expiry: an offer ends or Groupon deactivates it (expired), so it must stop being listed.
• Scope: new offers arrive in your inventory scope, and offers leave it when Groupon removes them or you narrow your scope.

Why a daily resync is the minimum

A delta only returns the offers whose own update time moved, and it cannot show you an offer that left your scope. When a promotion starts, many prices change at once without touching each offer's own update time: Groupon then ignores updatedSince for that walk and returns the whole catalog. So a delta alone cannot be trusted to have seen every change. A full walk is the one run that sees all of it, and the only one that lets you notice what is no longer there. Your listing must not stay wrong for longer than a day, so run the full walk every day, and let the delta close the gap in between. The cart endpoints re-check price and availability live, so a stale listing is corrected at add-to-cart time, but the shopper has already seen the wrong price by then.

Step 1: full sync, first

params = { country: "US", limit: 10 }               # no updatedSince
cursor = none
repeat:
    page = GET https://api-staging-core.livingsocial.com/partner_storefront/products?params (+ cursor if set)
    upsert every product in page.data by its id
    cursor = page.nextCursor
until page.hasMore == false
lastRefreshAt = page.timestamp - 10 minutes    # the final page's timestamp; persist ONLY after
                                                # the whole walk succeeds

Step 2: daily resync, every day

The same walk as step 1, run again against the database you already have, at a quiet hour. It also marks what the walk did not return:

seen = empty set
params = { country: "US", limit: 10 }               # no updatedSince
cursor = none
repeat:
    page = GET https://api-staging-core.livingsocial.com/partner_storefront/products?params (+ cursor if set)
    upsert every product in page.data by its id    # replace options, prices, status
    add every id in page.data to seen
    cursor = page.nextCursor
until page.hasMore == false
stop listing every product in your database whose id is not in seen    # keep the row, do not
                                                                       # delete it
lastRefreshAt = page.timestamp - 10 minutes    # persist ONLY after the whole walk succeeds

Stop listing products only after the whole walk has succeeded. If it fails part-way, change nothing and run it again.

Step 3: delta refresh, every two hours

params = { country: "US", limit: 10, updatedSince: lastRefreshAt }    # do NOT send active
cursor = none
repeat:
    page = GET https://api-staging-core.livingsocial.com/partner_storefront/products?params (+ cursor if set)
    upsert every product in page.data by its id    # replace options, prices, status
    cursor = page.nextCursor
until page.hasMore == false
lastRefreshAt = page.timestamp - 10 minutes    # the final page's timestamp; persist ONLY after
                                                # the whole walk succeeds

Store lastRefreshAt durably, as a database row. If a walk fails part-way, do not advance it: restart the walk from the previous lastRefreshAt.

The 10-minute overlap is intended. Upserts are idempotent, so re-reading an offer is harmless, and missing one is not.

Take the time from the final page's timestamp (Groupon's clock), not your own clock: an updatedSince in the future is rejected.

Do not send active=true on a delta. An offer that was deactivated since your last refresh comes back with a status other than active; you need that row to stop listing it.

After each upsert, recompute whether the product is listable (see "How to use the fields" above).

A delta can legitimately return the whole catalog. When a promotion starts, many prices change at once without touching each offer's own update time, so Groupon ignores updatedSince for that walk. The decision is made once, on the first page, and carried in the cursor, so a walk is never half incremental and half full. Your code must handle a delta of any size.

Do not run two walks at once: when a delta is due while a resync is running, skip the delta. You can also repeat the full sync at any time, for example to rebuild your database.