Sign inSell data
KEEP GOOD DATA FLOWING

Connect your collection pipeline.

Generate a scoped ingestion key in your dataset settings. Use it only in your server or scraper environment.

A stable record contract

{"id":"profile-123","followers":12800,"_version":2,"_op":"upsert"}

The primary key identifies the entity or event. Increase the integer _version when its source changes. A repeated identity and version must have identical business content. Use delete for a tombstone or correction for a free repair. Older source versions cannot overwrite newer data.

Connect in Python or JavaScript

Download the Python client or Node.js client. Both upload in bounded chunks, resume from a local checkpoint and wait for full validation. Keep the checkpoint to resume; remove it when deliberately starting a new source run. Set DATATAP_URL and DATATAP_INGEST_KEY in your server environment.

python ingest.py DATASET_ID data.ndjson --format ndjson
node ingest.mjs DATASET_ID data.ndjson ndjson

# After a successful source check with no changes:
python ingest.py DATASET_ID --no-changes

Corrections require a descriptive _correction_reason. CSV uses \\N for null; an empty string remains an empty string. Dates use ISO dates, timestamps include a timezone, and large identifiers or exact decimals should be strings. Your first completed upload is the connection test; check the job result before scheduling another run.

Resumable upload protocol

  1. POST /api/v1/seller/datasets/:id/uploads with action: start, file bytes, format and a UUID idempotency key.
  2. Request each signed part URL with action: part, upload ID and part number. PUT the file chunk to that URL and keep its ETag.
  3. POST action: complete, upload ID, and the ordered part numbers and ETags.
  4. The response contains a job ID. Full validation must finish before the new version is available.

Send Authorization: Bearer YOUR_INGESTION_KEY on the API requests. Each key can upload only to its assigned dataset. Accepted formats: CSV, NDJSON, JSON arrays and Parquet.

A successful check with no changes

POST https://datap.app/api/v1/ingest/DATASET_ID/check
Authorization: Bearer YOUR_INGESTION_KEY
Content-Type: application/json

{"key":"A_NEW_UUID_FOR_THIS_RUN","result":"no_changes"}

Send this after your scraper successfully checks its source and finds no changes. A process keepalive does not prove a source check. DataTap tracks source contact, successful checks and material changes separately.

Simple freshness rules

A source becomes overdue after its expected interval plus the greater of one hour or 25% of that interval. A weekly source has 8 days and 18 hours. Monitoring can be paused without deleting data or cancelling buyer access.