Comparison
Pull-based ingestionvsPush-based ingestion
Pull-based ingestion
you ask the source for new rows on a schedule, so an outage on your side just means the next run reads more.
You fetch from the source on your own schedule, tracking your own position. It is the resilient default — being down costs latency rather than data, and you control the rate — at the price of load on the source and latency equal to the poll interval. The choice between pushing and pulling is mostly a choice about who owns the failure, which is why it survives as an argument long after the latency numbers are agreed.
Full entry →Push-based ingestion
the vendor posts to your endpoint whenever something changes, and when your endpoint was down for an hour those changes are simply gone.
The source sends data to you as it happens, over a webhook or a stream write. It is the low-latency option and it moves the cost of delivery onto the sender, but it makes you responsible for being available: what the sender does when you are not — retry, drop, or nothing at all — is a property of their system, not yours. Read their retry policy before you rely on it, because most webhook providers give up quietly.
Full entry →