Sensor Accuracy, Resolution and Calibration Explained
difference in proximity to reference, repeatability and smallest numerical step; Educational example of offset and slope correction with Python code and independent test.
Contents
Sensors Laboratory • 18 Mehr 1405
difference in proximity to reference, repeatability and smallest numerical step; Educational example of offset and slope correction with Python code and independent test.
1. A beautiful display does not equal a good measurement
Suppose a display shows the temperature with several decimal places. The number of digits is only for display. If the sensor is far from the reference or changes a lot with each reading, adding decimals won't solve the problem. To evaluate, we need to look at both the distance from the reference and the scatter of the readings.
Metrology vocabulary in the BIPM reference distinguishes between the concept of accuracy and related concepts. In this article, we check "closeness to reference" with numerical error and measure "repeatability" under the same conditions. Resolution is also the smallest distinguishable change in the reading path; Do not summarize these three in one advertising number.
2. Three concepts in one experiment
| concept | practical question | common error |
|---|---|---|
| Proximity to the reference | How far is the average reading from the reference? | Consider more decimals as a sign of less error |
| Reproducibility | Under constant conditions, how scattered are the results? | Let's call the fixed but deviated output accurate |
| Resolution | What is the smallest separable step? | Let the number of bits be the accuracy of the whole system |
For comparison, the reference and sensor must actually see the same quantity. If one is attached to the body and the other is in the air, the difference in number is not necessarily a sensor error. Record the installation status, waiting time and environment conditions along with the result.
3. First, record the raw and reference data
Create a table containing time, raw data, transformed value, reference value and test conditions. Record several readings in each situation and separate transient behavior from sustained readings. Choose the number of samples and the time interval based on the rate of change of the quantity; There is no fixed number for all sensors.
The reference itself has limitations. If the comparator is just another monitor with an unknown error, call the result a "comparison of two devices", not a certified calibration. For sensitive work, the reference method and uncertainty should be specified according to the project requirement.
4. Educational model for correction of offset and slope
Calibration determines the relationship of the reading to the reference; Correcting or adjusting the device is another step. The following example is a training model for correction based on reference data and does not replace the full calibration method with uncertainty.
If the response is approximately linear in the desired range, two reference pairs can be used: the x₁ reading corresponding to the y₁ reference and the x₂ reading corresponding to the y₂ reference. The following coefficients are obtained from the line equation. This training model does not guarantee non-linear behavior or variation of error with temperature.
a = (y2 - y1) / (x2 - x1)
b = y1 - a * x1
y_corrected = a * x + bThe example is purely hypothetical: if 12 and 92 correspond to references 10 and 90 in the raw path, the slope is 1 and the offset is negative 2. A reading of 52 becomes 50 after correction. These numbers are not the test data of any real sensor.
5. the third point; A test that should not be omitted
Two points forming a line can be placed exactly on the same line. So success in these two points alone does not prove the validity of the model. Check one or more midpoints not used in the fit and keep the residual error.
If the mean error is greater than the project needs, narrow the interval or re-examine the model and measurement method. Do not extrapolate outside the range of two references without valid evidence. Also save the coefficients along with the part model, date, unit and installation conditions so that later it is clear for which assembly they were valid.
6. How is the code accompanying the article used?
The calibration_demo.py file only requires the Python 3 standard library. First, it runs the hypothetical example, then checks that two identical inputs do not cause division by zero. You can replace your reference pairs, but you must keep the units of both axes fixed.
The working example is an educational tool and does not produce a calibration certificate. Its output is suitable for understanding calculation and control of elementary errors. After substituting real data, keep an independent point for evaluation and don't accept coefficients just based on how good the graph looks.
7. The limit of software correction
Fixed offset, saturation, cable disconnection, and invalid sampling are not the same problem. Do not make a calibration factor from the saturation value or communication error. First, check the health of the data and enter only valid data into the model.
ADC calibration also has its own stage. The Espressif documentation describes methods for converting a raw value to a calibrated voltage; After that, the input circuit and the sensor must be evaluated separately. Correcting one step of the chain does not guarantee the whole machine.
Exercise files and the next step
Get the practice sheet and accompanying files (ZIP)
The files are prepared for the training exercise. Software testing does not take the place of testing your actual parts and wiring.
Frequently asked questions
Is an ADC with more bits necessarily more accurate?
no The number of bits is related to the numerical step; Noise, reference, circuit and sensor error also have an effect.
Are two points for each sensor enough?
no The linear model should be tested with independent points and in the range of application.
Does averaging remove constant offset?
no Averaging does not replace systematic error correction and reference checking.
Official sources
Technical resources reviewed on 10/10/2026. For hardware implementation, match the model and version of your part's documentation.
Question and answer article
Ask your technical question
Questions and answers are published after moderation. Authors can see their own pending drafts.
No confirmed questions have been registered for this article yet.