Coming soon. The key and the two options below, Keep data and Track changes, are set by a platform admin today. Self-serve settings, and a view of kept items and their past values, are on the way.

Every row a scrap returns is kept as an item. This guide explains what an item is, how Trawl tells one row from another, and what the two options change.

What an item is

One row in a run's results is one item. A scrap that returns 40 products gives you 40 items.

Items are replaced at each successful run: the items of a scrap match its latest successful run that passed the data quality check. A failed, empty or regressed run leaves them as they were. A row that is gone from the page is gone from the items, and a row whose values changed shows the new values. You never end up with duplicates after a re-run.

The key

The key is the field that identifies a row: the same row in two runs has the same key. Typical keys:

The key is one or more fields of your scrap's output. A run whose rows have a missing, empty or duplicated key value is marked failed, and the message names the row, so you can fix the selector.

Without a key, items work as described above: they follow the latest run. The two options below both need a key, because they have to recognise a row from one run to the next.

Keep data

By default, a row that leaves the results leaves the items.

With Keep data, items stay after they leave the results. A product that sells out and disappears from the page is still there, with its last known values.

Use it when you want a complete list over time rather than a mirror of the page.

Track changes

Track changes keeps the past values of the fields you choose, for each item.

A point is recorded only when the value changes. A price that stays at 19.90 for 30 days is one point, not 30. When it moves to 17.90, a second point is added, dated by the run that saw the change.

Use it for what moves: a price, a stock level, a rating. Pick only the fields you care about. A field inside a list (one price per variant, for example) needs extra setup to be tracked.

Without Keep data, an item and its history go away when its row leaves the results. To follow a product that can disappear, use both options.

Setting the options

The key and both options are set per scrap, and both options need a key.

Today a platform admin sets them. Self-serve settings, from the scrap's settings page, are coming soon.

Next steps

• Results payload: the shape of a run's rows.
• Data quality: spot a run that came back wrong.
• Scheduling: how often the items refresh.