Data source
What a data source does
A data source is a URL to your full product feed. Zaproo downloads it on a schedule, turns each item into a document and compares it with what the feed sent last time. Only new and changed products are written. Products that are no longer in the feed are deleted. You write no sync code.
Each index has at most one data source. The feed must hold the whole catalog on every download, because a product missing from the feed counts as deleted.
The data source writes through normal document tasks, so you can follow its writes on the Tasks page. Documents you send through the API are not touched unless the feed also holds their id.
Before you start
The data source writes into the fields of the index mapping, so the index needs a mapping first. An index created without fields gets one from its first upload, or you add the fields on Fields & mapping. Until then the Data source page links you there, and saving answers 409 FEED_NEEDS_MAPPING.
Set it up
Enter the feed
Open the index, then Data source. Enter the Feed URL, for example https://shop.example.com/feed.xml, and choose the format.
| Format | Items | Options |
|---|---|---|
| CSV / TSV | One row per item; the first row names the fields. | Delimiter (comma, semicolon, tab or pipe) and Encoding (utf-8, windows-1251, windows-1252). |
| JSON | The objects of one array. | Item path: a dotted path to the array, such as data.products. Leave it empty when the file is the array. |
| JSON Lines | One object per line. | None. |
| XML | One element per item. | Item path: element names joined by /. The default is rss/channel/item, which fits Google Shopping feeds. |
Any format may be gzipped. Zaproo detects it.
Preview
Click Preview. Zaproo reads the first 50 items and lists every field it found with sample values. Nothing is saved.
How fields are named:
- CSV: the header name.
- JSON: a dotted path, such as
price.amount. - XML: the child element name as written, prefix included, such as
g:price. Nested elements join with a dot, such asg:shipping.g:country. An element that repeats becomes a list.
Map the fields
For the id and for each index field you want filled, choose the source field. Leave a field Not mapped to skip it. Then click Preview mapped documents to see the first documents as they will be written, and the items that would be skipped.
Save
Choose the schedule and click Save. The first run starts within a minute.
Map the fields
Each index field takes its value from one source field. Two optional steps change the value first.
- Pattern is a regular expression with exactly one capture group. Only the captured part is kept. To take the id from a product URL like
https://shop.example.com/p/1042, choose the URL column and enter/p/(\d+). When the pattern does not match, the value is empty. - Split on turns a text into a list. With
,, the valuered, bluebecomes["red", "blue"].
Then the value is converted to the type of the index field:
| Field type | What is accepted |
|---|---|
float, double, integer, long | The first number in the text: 19.99 EUR becomes 19.99. A comma decimal such as 19,99 works when there is no dot. |
boolean | true, false, 1, 0, yes, no. |
date | ISO dates only, such as 2026-10-16 or 2026-10-16T10:00:00Z. |
hierarchy | Passed as is, such as Office > Paper > A4. |
text, keyword | Passed as is. |
An empty value leaves the field out of the document.
An item without an id, or whose document does not fit the mapping, is invalid. It is skipped, counted and listed with its reason in the run log. An invalid item is never treated as deleted.
If you later remove a field from the index mapping that the data source still fills, runs fail with FEED_MAPPING_OUT_OF_DATE until you update the data source.
Schedule
The plan sets how often a data source may run:
| Plan | Shortest interval |
|---|---|
| Free | Every day (1440 minutes) |
| Starter | Every 6 hours (360 minutes) |
| Growth | Every hour (60 minutes) |
| Enterprise | Every 15 minutes |
Sync now starts a run straight away. It is refused with 409 FEED_RUN_PENDING while a run is queued or running, and with 429 FEED_SYNC_TOO_SOON within 15 minutes of the previous Sync now.
What a run does
- Downloads the feed. Zaproo sends the
ETagandLast-Modifiedof the previous download. When the server answers304 Not Modified, the run ends asNOT_MODIFIEDand nothing is read. - Maps every item and compares it with the version the feed sent last time.
- Writes new and changed products as upsert tasks. Unchanged products are not written.
- Deletes the products that the data source wrote before and that are no longer in the feed. It never deletes documents it did not write.
A write that fails is retried on the next run.
The delete guard
A broken export can look like a catalog that lost most of its products. To protect the index, a run deletes nothing and ends as GUARDED when:
- more than 20% of the products the data source wrote would be deleted (change the share with Max deletions per run, or
maxDeletePercentin the API), or - more than 10% of the items in the feed are invalid.
New and changed products are still written. Once the feed is fixed, the next run deletes as usual. A feed with no valid items fails with EMPTY_FEED and deletes nothing.
Authentication
Choose how Zaproo signs in to your feed:
- None.
- Basic: enter
user:password. - Header: enter one header as
Name: value, such asX-Api-Key: 3f9c….
The secret is stored encrypted and never shown again. The page shows that a secret is stored. Leave the box empty to keep it, or type a new one to replace it.
The URL must be public http or https. Addresses in private and local networks are refused.
Run log
The Run log lists the runs in two tabs. Succeeded holds SUCCEEDED and NOT_MODIFIED runs. Failed holds FAILED and GUARDED runs. A run in progress shows on top. Click a run to see the HTTP answer, the counts (seen, created, updated, unchanged, deleted, invalid, duplicates), the error, up to 50 skipped items with their reason, the start of the response body for failed and guarded runs, and links to its tasks.
Zaproo keeps the newest 20 successful and the newest 20 failed runs.
After 10 failed runs in a row the data source turns itself off. The page shows the last error. Fix the cause, switch Enabled back on and save.
Integrations
A shop platform plugin, such as the Magento 2 module, writes documents itself, so an index it is connected to does not need a data source. While a plugin is connected, a data source cannot be saved or synced (409 INDEX_HAS_INTEGRATION) and does not run on its schedule. A data source saved before stays stored and resumes when the plugin is disconnected. You can still view it, preview it, read its run log and delete it.
The plugin connects with a full-access secret key and reports its settings each time they change. The index overview and the Data source page show it. To stop using it, for example when the shop is gone or the plugin is uninstalled, click Disconnect there. Documents stay in the index. A plugin that is still installed connects again the next time it saves its settings.
PUT/v1/projects/:projectId/indices/:indexId/integrationA full-access secret keyDELETE/v1/projects/:projectId/indices/:indexId/integrationProject admin or a full-access secret keyErrors
| Code | When |
|---|---|
409 FEED_NEEDS_MAPPING | The index has no mapping yet. |
409 INDEX_HAS_INTEGRATION | Saving or Sync now while a shop platform plugin is connected. See Integrations. |
400 FEED_MAPPING_INVALID | A mapped index field is not in the index mapping. |
400 FEED_INTERVAL_NOT_ALLOWED | The interval is shorter than the plan allows. |
400 FEED_AUTH_SECRET_REQUIRED | Basic or Header authentication was chosen without a secret, and none of that kind is stored. |
422 FEED_FETCH_FAILED | Preview could not download or read the feed. The message says why. |
409 FEED_RUN_PENDING | Sync now while a run is queued or running. |
429 FEED_SYNC_TOO_SOON | Sync now within 15 minutes of the previous one. |