The "Store and Forward" Reality

In an ideal world, IoT devices maintain a persistent connection, streaming telemetry in real-time via MQTT or WebSockets. In the real world-especially in industrial or environmental monitoring-connectivity is a luxury, not a guarantee.

Whether it's a GSM blackout in a rural hydro-station or a battery-saving sleep cycle, devices frequently operate in "Store and Forward" mode. They buffer data locally and flush it in bulk once connectivity is restored.

For engineers integrating these devices via standard REST APIs, this behavior creates a critical synchronization challenge. If your ingestion strategy relies on "current time," you are already losing data.

The Trap: Naive Time-Based Polling

The most common mistake in IoT ingestion is coupling the data fetch window to the server's clock. Imagine a cron job running every hour to fetch data from the "last 60 minutes."

GET /measurements?from=now-1h&to=now

Consider this scenario: at 10:00 AM the device loses GSM coverage. At 12:00 PM it regains signal and uploads buffered data from 10:00 to 12:00. At 12:05 PM your cron job runs - it asks for data from 11:05 to 12:05.

The data from 10:00 to 11:05 is successfully uploaded to the manufacturer's cloud, but your system never requested it. It fell outside the sliding window. These gaps are often silent, discovered only months later during historical analysis. By then, automated alerts for water leaks fail to trigger, predictive maintenance models are fed corrupted data, and compliance reports become legally invalid. In IoT, silent data loss is a silent revenue leak.

The Solution: Watermark Strategy

To achieve 100% data integrity, we must decouple "ingestion time" from "event time." The Watermark Strategy (also known as Cursor-based synchronization) relies on the state of our database, not the clock. Instead of asking "What happened in the last hour?", the system asks: "What is the timestamp of the last record I successfully saved?"

The fetch logic transforms from a sliding window to an anchored cursor:

const watermark = await db.measurements.findLast().timestamp;

GET /measurements?from=${watermark}

By using the last known data point as the anchor, the system automatically backfills any historical gaps. If a device goes offline for three days, the next sync request will naturally expand its window to cover those three days - with zero manual intervention.

Implementation Logic

At Silentbits, we implement this using a robust scheduler pattern. Below is a simplified logic flow for a Node.js ingestion service:

async function syncStationData(stationId: string) {
  // 1. Retrieve the "Watermark" (last successful sync time)
  const lastRecord = await prisma.measurement.findFirst({
    where: { stationId },
    orderBy: { timestamp: 'desc' }
  });

  // Default to 24h ago if new station, else use Watermark
  const fromDate = lastRecord
    ? lastRecord.timestamp
    : subHours(new Date(), 24);

  // 2. Fetch data starting from Watermark
  const data = await api.fetchMeasurements({
    stationId,
    from: fromDate.toISOString()
  });

  if (data.length === 0) return; // System is up to date

  // 3. Persist and update state
  await prisma.measurement.createMany({ data });
  console.log(`Synced ${data.length} records. Gap closed.`);
}

REST vs. MQTT: Architectural Context

It is important to acknowledge that this pattern is a specific solution for PULL-based integrations (REST APIs). In modern, native IoT architectures where we control the firmware, we prefer PUSH-based protocols like MQTT or CoAP. These protocols handle QoS (Quality of Service) at the transport layer, ensuring message delivery without custom syncing logic.

However, in the enterprise world, we often integrate with third-party hardware - weather stations, legacy PLCs - where a manufacturer's REST API is the only interface. In these environments, the Watermark Strategy is not just an option. It is the industry standard for reliability.