Connecting a device: designing the data path from sensor to cloud

A connected device is only as useful as the data that reaches the people who rely on it. On the way from a sensor to a chart in an app, that data passes through firmware, a radio, perhaps a phone or a gateway, a network and a cloud service, and each step can delay it, lose it or duplicate it.

Many of the hard problems sit at the boundaries between hardware, firmware, app and cloud: a timestamp that one side assumes and the other never sets, a retry that creates duplicates, an update that sends a format the cloud cannot yet read. Designing the path as one system from the start avoids most of them.

Start from the data

Before choosing a radio or a cloud service, describe the data. The answers drive almost every later decision:

• What is measured. Each quantity, its range and resolution, and whether it is a continuous signal, a periodic reading or an event such as an alarm.
• How often. A sensor often has to be sampled far faster than anyone needs to see results, which leaves room for processing on the device.
• How fresh it must be. Some data must reach the app within moments; much of it can arrive later, in batches. This decides whether the radio can sleep for long periods.
• What may be lost. Whether a gap is acceptable, and whether an old record is still worth delivering once newer ones exist.

Multiplying the size of each record by the rate of records gives the volume to send. If it is far beyond what the radio can carry or the battery can afford, the device has to reduce the data first.

Choosing the wireless link

No single radio suits every device. Weigh the range needed, through walls, metal and people; the data rate, including peaks such as a firmware download; the power, since radios that reach further or carry more data generally use more energy; and the infrastructure that must already exist where the device is used.

Direct, through a phone or through a gateway

• Bluetooth Low Energy through a phone. The radio is efficient and the phone provides the internet connection, but data flows only while the phone is nearby and the app is allowed to run in the background.
• Wi-Fi. Connects directly to the internet through an existing network and carries plenty of data, but generally uses more power than Bluetooth Low Energy, and the device needs the network's credentials during setup.
• Cellular. Works wherever there is coverage, at the cost of higher power use, a subscription for each device and more complex hardware.
• Low-power wide-area networks. Reach long distances on very little power, but carry only small, infrequent messages and need a gateway or a network operator covering the area.
• A local gateway. Collects data from many nearby devices over a short-range link and forwards it over Wi-Fi, cellular or a wired connection.

Test the link early, where and how the device will be used: range figures in datasheets describe ideal conditions.

On the device

Sampling and local processing

Sample each sensor as fast as the measurement needs, and no faster. Where a sensor can buffer readings on its own, let the processor sleep until a batch is ready or a threshold is crossed.

Processing on the device can greatly reduce what is sent: summaries over an interval, events instead of a stream of samples, and a compact binary encoding. Raw data that is never sent cannot be recovered, so reduce it deliberately, and record which firmware version produced each record.

Buffering while offline

Every connected device spends time without a connection. Treat it as normal operation, and decide from the data specification how much of the data must survive it:

• Where records must not be lost, keep each one in non-volatile memory until the cloud confirms receipt, not merely until it has been sent.
• Where only recent values matter, a buffer in RAM, or keeping only the latest value, may be enough, and it spares both the flash memory and the battery.
• Size the storage for the longest offline period the application must cover, decide what to discard when it is full, and record that data was dropped.
• Flash memory wears with repeated writes, and an interrupted write can corrupt it. Spread writes across the memory, and use a layout that survives a power cut at any moment.
• Decide whether a backlog is sent newest first, to keep a live view current, or oldest first, to keep the history in order.

Timestamps and time

Timestamp each record on the device at the moment of measurement: once records are buffered, the time of arrival says little about when they were measured. Every clock drifts, and one that loses power restarts from an arbitrary value, so:

• Synchronize the clock whenever a connection allows, and record when that last happened.
• Store timestamps in one fixed time reference, never local time, and convert only for display.
• Without a valid time, record a counter since startup together with a restart count. If the device later synchronizes its clock without restarting, the cloud can convert those counters to times, with an error that grows with the clock's drift over the interval. Records from a run that ended before any synchronization, for example through a power loss, cannot be placed in time reliably: keep their order, and mark their time as unknown or estimated. A real-time clock kept running by a backup supply avoids most of this.
• Mark records whose time is uncertain, rather than giving them a wrong one.

Designing the messages

The message format is a contract that outlives any firmware version: old firmware keeps running in the field long after a new version is released, so the cloud must understand every version still in use.

• Keep messages small. A compact binary format uses fewer bytes than text, saving energy and airtime.
• Version every message. Design new versions so that older readers can skip fields they do not recognize.
• Number every record. A device identifier, a restart count and a sequence number within each run let the cloud detect gaps, discard duplicates and restore order. Store the restart count in non-volatile memory once per start, and keep the sequence number in RAM: writing a counter to flash for every record wears the flash out. If one continuous number is needed, reserve numbers in blocks: store only the start of the next block, and continue from it after a restart.
• Acknowledge and retry. Resend each record until the receiver acknowledges it. Lengthen the delay after each failed attempt and add a random element, so that devices recovering from the same outage do not all retry at once.
• Make duplicates harmless. Store records keyed by device and sequence number, so that a record delivered twice is stored once.
• Expect any order. Order data by timestamp and sequence number, never by arrival.

Commands to the device need the same care, plus an expiry, so that a late command is not carried out after it has lost its meaning. A lightweight publish-subscribe protocol keeps a connection open so the cloud can send commands at any time; short requests let the device sleep between them but pay the cost of connecting each time.

Power and the radio

In many battery-powered devices, the radio is among the largest users of energy, and waking it, joining the network, setting up an encrypted session and waiting for acknowledgments can cost far more than sending the data.

• Batch transmissions. One connection carrying many records spreads the fixed cost of connecting, within the freshness each kind of data requires.
• Set the duty cycle deliberately. Derive how often the device connects and listens from the freshness requirements, and let the cloud adjust it once real usage is known.
• Back off after failures. Repeated attempts to connect where there is no coverage can drain a battery quickly.
• Plan for current peaks. The supply must deliver the radio's transmit peaks without its voltage dropping far enough to reset the device, even from a cold or nearly empty battery.

Security from the first design

A connected device stays reachable for as long as it is in use, and a weakness found after release must be fixed on every unit in the field. Security is far easier to design in than to add later.

• A unique identity for each device. Its own key, created or installed before the device ships and kept in protected memory or a secure element, so that a key extracted from one device opens no other.
• No shared default credentials. Never ship devices that share a password, key or setup code.
• Encrypted, authenticated connections. Encrypt every link, with the device and the server each verifying the other, and pair with a phone securely, never over an open connection.
• Secure boot and signed updates. The device checks the signature of its firmware before running it and of each update before installing it. A signature alone does not stop an older, correctly signed version with a known weakness from being installed, so the device can also keep a minimum version, stored where it can only be increased, and refuse any image older than it. Raise the minimum when older versions must no longer run, and only after the new version has confirmed that it works, so that the automatic rollback described below can still return to the previous image if the update fails.
• Closed debug access. Disable or lock the debug and programming interfaces on the units that ship.
• Least privilege in the cloud. Each device may send only its own data and receive only its own commands, and each credential can be revoked on its own.

Collect only the data the product needs, encrypt it where it is stored, and decide how long it is kept.

Updates in the field

Over-the-air updates deliver fixes after a device ships, and they are the riskiest thing it does: a failed update can leave a device that no longer starts and is out of reach.

• A layout planned for failure. Three common designs are alternatives, each with its own cost. With two firmware slots, often called A/B, the device downloads into the inactive slot while it keeps working, verifies the image, then switches; this roughly doubles the memory needed for firmware. A staged copy keeps the new image in separate storage and lets the bootloader copy it into place in a way that resumes after a power cut. A small recovery image can download the firmware again when the main image fails. None of them makes an update safe on its own: each depends on its bootloader, on verifying the image and on testing, including power cuts at every step. Choose the layout when choosing the memory.
• Automatic rollback or recovery. The new firmware must confirm that it works, for example by reconnecting to the cloud; otherwise the bootloader returns to the previous image, or to the recovery path where no previous image is kept.
• Resumable downloads and safe conditions. Resume interrupted downloads, and install only with enough battery charge and no critical task running.
• Staged rollout. Release to a small group of devices first, watch for failures, then widen the release step by step, with the ability to pause.
• Compatibility. Every firmware version in the field must be able to receive the next update.

In the cloud and the app

Ingestion and storage

The ingestion service authenticates each message, checks its format and version, stores it once and acknowledges it. Keep it simple, leave heavier processing to later steps that can be run again if they fail, and expect bursts when many devices reconnect after an outage.

Store measurements keyed by device and time of measurement, and keep the raw records with their firmware version and arrival time, so that processing can be corrected and repeated. Keep status reports and logs, such as battery level, signal strength and recent errors, separate from the measurements.

Presenting the data

An app that shows only the latest value hides the most important fact about connected data: its age. Show when each value was measured, mark values that are out of date, and show gaps as gaps rather than drawing a line across them. Distinguish a device that was offline from a reading of zero.

Testing the whole path

Most failures of connected devices appear under conditions that are rare on the bench and common in the field. Test the whole path, from sensor to screen, under those conditions:

• Poor and intermittent connectivity. Weak signal, links that drop during a transfer, a phone that leaves and returns, and a network that connects but passes no data. A network emulator or a shielded enclosure makes these repeatable.
• Long offline periods. Fill the storage, reconnect, and check that the backlog arrives complete, in order and without duplicates.
• Power loss. Cut power during storage writes, transmissions and updates, and check that the device recovers with its data intact.
• Clock drift. Start devices with no valid time, let clocks drift and change the time abruptly, then check the timestamps in the cloud.
• Many devices at once. Simulate many devices connecting together, including after an outage.
• Mixed versions. Run old and new firmware, app and cloud versions together, including updates and rollbacks.

Keep some devices running for long periods: slow problems show up only over time.

A checklist for connected devices

• Each kind of data has a written rate, size, freshness requirement and tolerance for loss.
• The wireless link suits the product's range, data rate, power and infrastructure, and has been tested where the device will be used.
• Records that must not be lost stay on the device until the cloud acknowledges them.
• Every record carries its time of measurement, and the device synchronizes its clock whenever it can.
• Messages are compact, versioned, numbered and safe to retry.
• The energy of a complete connection is measured, and transmissions are batched to fit the power budget.
• Every device has its own identity, links are encrypted and authenticated, and firmware is signed and verified.
• Updates are verified before they run, are designed and tested to survive a power cut at any point, roll back or recover when the new firmware fails, and are released in stages.
• The app shows the age of each value and any gaps.
• The whole path has been tested with poor connectivity, long offline periods, clock drift, power loss and many devices at once.