Machine Learning Foundations / Beginner / 01.45
IoT: data from sensors
Sensors measure the real world and send numbers to a computer. That makes them a rich source of data for machine learning, and a messy one.
14 min readFree
What the Internet of Things is
In the last lesson we met devices as one source of data. This lesson goes deeper, because sensor data behaves differently from a spreadsheet someone typed. A sensor keeps measuring and sending numbers, whether they are useful or not.
IBM defines the Internet of Things, or IoT, as 'a network of physical devices, vehicles, appliances, and other physical objects that are embedded with sensors, software, and network connectivity, allowing them to collect and share data.' Put simply, it is everyday objects that can measure something and talk over a network. IBM says these devices 'can range from simple "smart home" devices like smart thermostats, to wearables like smartwatches and RFID-enabled clothing, to complex industrial machinery and transportation systems.'
Why does this matter for machine learning? A model needs examples, and devices produce them all day long. IBM notes that in a business, IoT devices 'are used to monitor a wide range of parameters such as temperature, humidity, air quality, energy consumption, and machine performance.' Each of those is a stream of numbers we could learn from.
Key words
- IoT (Internet of Things)
- A network of physical objects with sensors, software and a network connection, so they can collect and share data.
- Sensor
- A part that measures something in the world and reports it. Wikipedia defines it as a device that 'detects events or changes in its environment and sends the information to other electronics, frequently a computer processor.'
- Reading
- One measurement from one sensor at one moment.
- Time series
- Wikipedia: 'a sequence of data points indexed, listed, or graphed in chronological order.' In short, readings in time order.
- Sampling rate
- How many readings the sensor takes each second.
- Broker
- A middle program that receives messages from devices and hands them to whoever asked for them. This lesson uses the word for what the MQTT standard calls the 'Server'.
Sensors and what they send
A sensor turns something physical into a number. Wikipedia says the stimulus 'is the quantity, property, or condition that is sensed and converted into electrical signal.' Software turns that signal into a number such as 21.4.
A bare '21.4' is not enough to learn from. Which room? When? Celsius or Fahrenheit? A useful reading carries a little extra information.
Timestamp
The date and time the reading was taken. Without it, the order of readings is lost, and order is the whole point of a time series.
Device ID
A name for the sensor, such as kitchen-1. It lets us tell one sensor's readings from another's.
Value and unit
The number itself, and what it counts, such as 21.4 and C for Celsius. A value without a unit is a guess.
Together those make one row. Many rows from one device make a time series, and Wikipedia adds that 'most commonly, a time series consists of observations recorded at successive equally spaced points in time.' Real sensors do not always keep to that spacing.
Here is the kind of data we will clean later in this lesson. Seven readings from a made-up kitchen sensor, one per minute. Does anything look wrong?
09:00 kitchen-1 21.0 C
09:01 kitchen-1 21.2 C
09:02 kitchen-1 (missing)
09:03 kitchen-1 21.5 C
09:04 kitchen-1 48.0 C
09:05 kitchen-1 21.7 C
09:06 kitchen-1 21.9 CTwo things stand out. The 09:02 reading never arrived, and 48.0 degrees in a kitchen, one minute after 21.5, looks wrong. A model that learns from such data without checking will learn the problems too.
Sampling rate: how often to measure
A sensor cannot watch the world every instant. It looks at it every so often. Wikipedia describes sampling as measuring a signal 'every T seconds, which is called the sampling interval or sampling period.' It then defines the sampling rate as 'the average number of samples obtained in one second.' One reading every 60 seconds is a long interval and a very low rate. A reading every tenth of a second is a short interval and a high rate.
Choosing the rate is a trade. A fast rate catches quick changes, but makes more data to send, store and clean. A slow rate is cheap, but a spike that lasts two seconds can vanish from a sensor that reports once a minute.
Tip
Write down the sampling rate beside the dataset. It decides which problems the data can show.
How the data travels
A reading starts on a small device and must reach a place where it can be stored. IBM says IoT lets smart devices 'communicate with each other and with other internet-enabled devices. Like smartphones and gateways'. In this lesson, a gateway is a box between the sensors and the wider network. It collects the readings and passes them on.
Many small devices use a messaging protocol called MQTT to send their readings. A protocol is an agreed set of rules. The MQTT website describes it this way: 'MQTT clients are very small, require minimal resources so can be used on small microcontrollers.' The official standard, published by OASIS, calls MQTT 'a Client Server publish/subscribe messaging transport protocol' that is 'light weight, open, simple'.
Publish and subscribe is the key idea. Think of a school notice board. Anyone can pin a note under a heading, such as 'Lost property', and anyone who cares reads that heading. The people who pin notes never need to know who reads them.
In MQTT, the standard calls a program or device that uses it a Client. A Client 'publishes Application Messages that other Clients might be interested in' and 'subscribes to request Application Messages that it is interested in receiving.' The middle program, which the standard calls the Server, is described as 'an intermediary between Clients which publish Application Messages and Clients which have made Subscriptions.' We call that middle program a broker. Each message carries 'payload data, a Quality of Service (QoS), a collection of Properties, and a Topic Name.' The topic name is the heading on the note.
- 1
1. A sensor takes a reading
The kitchen sensor measures 21.4 degrees.
- 2
2. It publishes to a topic
The device sends the number to the broker under a topic name of its own choosing, for example one that names the room and the kind of reading.
- 3
3. The broker passes it on
The broker holds the connections and forwards the message to every client that subscribed to that topic.
- 4
4. A subscriber receives it
A data collector, a dashboard or a machine learning pipeline gets the reading and can store it for training.
Networks fail, especially wireless ones. The MQTT website notes that IoT devices often use cellular links, and says 'Many IoT devices connect over unreliable cellular networks.' To cope, MQTT has three delivery levels: '0 - at most once, 1- at least once, 2 - exactly once'. 'At least once' means a message may arrive more than once, so check for duplicate readings.
Quick check
In the notice board picture, what plays the part of the heading on each note?
Edge or cloud: where to process it
Where should we work on readings? One choice is the cloud, big shared computers in data centres. The other is the edge. Wikipedia says edge computing 'brings computation and data storage closer to the sources of data', and that it aims to 'reduce the latency compared to when an application runs on a centralized data center.' Latency is the wait between asking and getting an answer.
Wikipedia adds that storing all IoT data in cloud data centers 'pushes network bandwidth requirements to the limit.' Bandwidth is how much data a network connection can carry. So a project may clean and shrink data at the edge, then send only the summary on.
The edge has a privacy benefit and a cost. Wikipedia says that 'by keeping and processing data at the edge, it is possible to increase privacy by minimizing the transmission of sensitive information to the cloud.' It also warns that edge nodes 'may also be resource-constrained devices, limiting the choice in terms of security methods.' Small devices cannot always run strong protection.
Common problems with sensor data
Noise and bias
Wikipedia says sensors 'cannot replicate an ideal transfer function', so deviations limit their accuracy. One is a sensitivity error. Another is an offset: 'If the output signal differs from the correct value by a constant, the sensor has an offset error or bias.'
Resolution
Wikipedia: 'The sensor resolution or measurement resolution is the smallest change that can be detected in the quantity that is being measured.' It adds that 'a sensor's accuracy may be considerably worse than its resolution.'
Missing readings
A reading may never arrive, because a battery died or a network dropped. The Python documentation notes that 'some datasets use NaN (not a number) values to represent missing data.'
Four more are habits to check, and are our own advice rather than facts from one source. Outliers are readings far from all the others, like the 48.0 in our kitchen: they may be real events or faults, so look before you delete. Clocks: every device has its own, so check that timestamps share one clock and one time zone, or readings will pair up wrongly. Units: keep the unit beside the value and convert to one unit before use. Drift: a slow change in what a sensor reports while the world stays the same, so compare old and new readings in steady conditions.
Tip
Plot your readings before you do anything else. Gaps, spikes and flat lines are easy to spot with your eyes, and much harder to spot in a table of numbers.
Cleaning the readings
Now we fix the kitchen data: find the gap, flag the outlier, smooth the rest, using only Python's standard library.
- 1
1. Make rows
Build one row per reading: timestamp, device ID, value and unit. None stands for the reading that never arrived.
- 2
2. Find the gap
List the times where the value is missing.
- 3
3. Flag the outlier
Take the median of the known values. The median is the middle value when they are sorted. Flag any reading more than 10 degrees away from it. The 10 is our own choice for this kitchen, not a rule.
- 4
4. Smooth
Remove the flagged reading, then average each run of three neighbouring readings.
from datetime import datetime, timedelta
from statistics import median
start = datetime(2026, 11, 16, 9, 0)
values = [21.0, 21.2, None, 21.5, 48.0, 21.7, 21.9]
readings = [(start + timedelta(minutes=i), "kitchen-1", v, "C") for i, v in enumerate(values)]
for ts, device, value, unit in readings:
print(ts.strftime("%H:%M"), device, value, unit)
missing = [ts.strftime("%H:%M") for ts, _, v, _ in readings if v is None]
print("missing at:", missing)
known = [v for _, _, v, _ in readings if v is not None]
mid = median(known)
outliers = [v for v in known if abs(v - mid) > 10]
print("median:", mid)
print("outliers:", outliers)
clean = [v for v in known if v not in outliers]
print("clean:", clean)
smooth = [round(sum(clean[i:i + 3]) / 3, 2) for i in range(len(clean) - 2)]
print("3-point moving average:", smooth)Output
09:00 kitchen-1 21.0 C 09:01 kitchen-1 21.2 C 09:02 kitchen-1 None C 09:03 kitchen-1 21.5 C 09:04 kitchen-1 48.0 C 09:05 kitchen-1 21.7 C 09:06 kitchen-1 21.9 C missing at: ['09:02'] median: 21.6 outliers: [48.0] clean: [21.0, 21.2, 21.5, 21.7, 21.9] 3-point moving average: [21.23, 21.47, 21.7]
The script found the gap at 09:02 and flagged 48.0. We used the median, not the average, because one wild value drags an average a long way but barely moves a median.
A moving average smooths a series. Wikipedia describes it as 'a calculation to analyze data points by creating a series of averages of different selections of the full data set.' It is 'commonly used with time series data to smooth out short-term fluctuations and highlight longer-term trends or cycles.' Our 3-point version averages each reading with its two neighbours, so five clean readings gave three smoothed ones.
Tip
Smoothing hides spikes. If spikes are what you want to predict, flag and keep them.
Filling gaps carefully
Should we fill the hole at 09:02? Maybe, but be careful. Wikipedia describes interpolation as 'estimation of an unknown quantity between two known quantities', and says it is useful 'where the data surrounding the missing data is available'. A filled value is a guess, not a measurement. A model that trains on a guess treats it as truth.
A safe habit is to fill only short gaps, with a rule you can state in one line. Below, a single missing reading becomes the average of its two neighbours, and longer gaps stay empty.
values = [21.0, 21.2, None, 21.5, None, None, 21.9]
filled = list(values)
for i in range(1, len(values) - 1):
one_gap = values[i] is None and values[i - 1] is not None and values[i + 1] is not None
if one_gap:
filled[i] = round((values[i - 1] + values[i + 1]) / 2, 2)
print("before:", values)
print("after: ", filled)Output
before: [21.0, 21.2, None, 21.5, None, None, 21.9] after: [21.0, 21.2, 21.35, 21.5, None, None, 21.9]
The single gap became 21.35. The two-reading gap stayed empty, because we have no honest basis for inventing two numbers. Keep a note of which values were filled.
Resampling is the other common job: devices report at different rates, so you pick one rate, such as one value per minute, and average each minute's readings into one. Decide the rule first and write it down.
Quick check
In the kitchen data, why did we use the median, not the average, to spot the outlier?
What teams do with it
Clean sensor data feeds many projects. IBM gives one reason businesses collect it: 'IoT sensors can be used to monitor equipment performance and detect or even resolve potential issues before they cause downtime, reducing maintenance costs and improving uptime.' We can call this predictive maintenance: a model could learn what normal readings look like for a motor, then warn when they change.
IBM lists 'energy consumption' among the parameters devices monitor, so a smart meter could give a steady stream for a model to learn a home's daily pattern. IBM also names smartwatches as IoT devices, so wearables could feed models too. Those readings are about a person, so the next section matters most for them.
Privacy and security
IBM puts security and privacy first among the risks of IoT: 'Many IoT devices are vulnerable to hackers and other cyberthreats, which can compromise the security and privacy of sensitive data. IoT devices can also collect vast amounts of personal data, raising concerns about privacy and data protection.'
There are protections to look for. The MQTT website says the protocol 'makes it easy to encrypt messages using TLS and authenticate clients using modern authentication protocols, such as OAuth.' Encrypting scrambles a message so only the right receiver can read it; authenticating proves who a client is.
Tip
Collect only the readings your project needs, and keep personal ones out of datasets you share. Data you never stored cannot leak.
What to remember
Each reading is a timestamp, a device ID, a value and a unit, and a run of them is a time series. Before you trust it, look for gaps, outliers, bias, clock and unit mix-ups, and drift. Fill gaps only when you can explain why, and keep a note of what you changed.
Next: data formats. Readings have to be written down somehow before they can be stored and shared, and the format we choose decides how easy they are to read, send and clean.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.