Use repeatability when the machine returns to taught or closed-loop positions, accuracy (non-linearity) when the reading must match an absolute dimension on a drawing, and resolution when the controller needs small position steps for smooth motion. Read all three in millimetres for your stroke, not only in % FS, before comparing sensors.
Position sensor datasheets list several precision figures side by side, and they are easy to mix up. This page defines each term as metrology uses it, converts the % FS values of the SEGMENsensor magnetostrictive position sensor series into micrometres for three stroke lengths, and shows which figure matters for which job. All product values come from the published series datasheets; values marked "calculated" are arithmetic from those figures, not measurements.
What Is the Difference Between Resolution, Accuracy and Repeatability?
Resolution is about the size of the smallest step, repeatability about scatter, and accuracy about distance from the true value. NIST defines accuracy of measurement as the closeness of agreement between a measurement result and the value of the measurand, and repeatability as the closeness of agreement between successive measurements of the same measurand under the same conditions (NIST TN 1297, Appendix D1 — Terminology ↗). The definitions follow the International Vocabulary of Metrology (VIM), BIPM JCGM Working Group 2 ↗.
| Term | Question it answers | How datasheets state it | Main limits |
|---|---|---|---|
| Resolution | What is the smallest position change the output can show? | µm, or bits for analog outputs (16-bit = 65,536 steps) | Signal electronics, output type, input card of the PLC |
| Repeatability | Do I get the same reading each time I return to the same position? | ±% FS or ±µm (often with a minimum value) | Noise, mechanical play, temperature changes during the test |
| Accuracy | How far is the reading from the true position? | Usually as non-linearity (±% FS) at reference conditions, plus temperature coefficient | Non-linearity, zero and span offset, temperature, mounting |
| Non-linearity | How far does the output curve deviate from a straight line over the stroke? | ±% FS, sometimes with a minimum in µm | Waveguide and calibration; the largest part of the accuracy budget at reference conditions |
Why Can a Sensor Resolve 0.1 µm but Only Be Accurate to ±50 µm?
Because resolution only describes the step size of the electronics, while accuracy adds every error between the true magnet position and the output. A magnetostrictive sensor measures the travel time of a torsional pulse along its waveguide (see how a magnetostrictive position sensor works). The time can be counted in very small increments, but small variations along the waveguide, the zero and span setting, temperature and the magnet mounting all shift the reading by more than one step. That is why each SEGMENsensor position sensor is calibrated at 10 points over its stroke, and why non-linearity, not resolution, is the figure to use for absolute accuracy.
How Do You Convert % FS Specifications into Micrometres?
Multiply the percentage by the full-scale stroke, then apply the minimum value if the datasheet gives one:
Error (µm) = max( %FS ÷ 100 × stroke (mm) × 1000 , minimum value )
Analog step (µm) = stroke (mm) × 1000 ÷ 216 for a 16-bit D/A output
Worked values for the SGM-PR10 / PE10 family (non-linearity <±0.01% FS, min ±50 µm; repeatability <±0.001% FS, min ±5 µm; 16-bit D/A). All figures in the table are calculated.
| Stroke | Non-linearity | Repeatability | 16-bit analog step |
|---|---|---|---|
| 300 mm | 0.01% × 300 mm = 30 µm → minimum applies: ±50 µm | 0.001% × 300 mm = 3 µm → minimum applies: ±5 µm | 300 000 ÷ 65 536 ≈ 4.6 µm |
| 1000 mm | ±100 µm | ±10 µm | ≈ 15.3 µm |
| 2500 mm | ±250 µm | ±25 µm | ≈ 38.1 µm |
Two lessons from the table. On short strokes the minimum value, not the percentage, sets the figure. On long strokes with an analog output, the analog step can be coarser than the repeatability, so a digital output such as SSI or RS485 keeps more of the sensor's resolution. In current terms, 16 bits over a 4–20 mA span is 16 mA ÷ 65,536 ≈ 0.24 µA per step (calculated); the PLC's analog input card must resolve at least that finely, or it becomes the limit. Check the scaling with the 4–20 mA calculator.
Which Specification Matters for Your Application?
Pick the figure that matches how the position is used: returning to the same point, matching a drawing, or controlling motion in small steps.
| Application | Spec to check first | Why |
|---|---|---|
| Hydraulic cylinder with taught end or stop positions | Repeatability | The controller compares with positions it learned from the same sensor, so a fixed offset cancels out |
| Closed-loop position control of a press or injection unit | Repeatability and resolution | Smooth control needs small steps; stable stops need low scatter |
| Reading must match an absolute dimension (gauging, test rigs) | Non-linearity (accuracy) | Every millimetre of the stroke must be right, not just repeatable |
| Long stroke with 4–20 mA or 0–10 V output | Analog step size | The 16-bit step grows with stroke; consider SSI or RS485 |
| Short stroke, fine measurement | Minimum values in µm | The ±50 µm / ±5 µm floors apply below 500 mm stroke (calculated) |
How Are Repeatability and Accuracy Tested?
- Repeatability: move the magnet to the same position several times from the same direction under unchanged conditions and record the spread of readings.
- Non-linearity: compare the output with a reference length standard at points spread over the stroke and find the largest deviation from the best straight line; SEGMENsensor calibrates position sensors at 10 points.
- Resolution: move the magnet in very small increments and find the smallest increment that changes the output.
Reference conditions and general test procedures for industrial measurement transmitters are set out in IEC 62828-1 — Reference conditions and procedures for testing industrial and process measurement transmitters ↗. For how % FS and % of reading differ on pressure instruments, see pressure transmitter accuracy: % FS vs % of reading.
- Magnetostrictive resolution: down to 0.1 µm (highest available; per model and output).
- SGM-PR10 / PE10 / PR20 / PE20 / PF10: non-linearity <±0.01% FS (min ±50 µm), repeatability <±0.001% FS (min ±5 µm), 16-bit D/A analog resolution.
- SGM-PF20 / PR40 / PR50: repeatability ±0.005 mm.
- Range: 25–8000 mm (SGM-PR10); in-cylinder pressure tube 35 MPa continuous.
- Outputs: 4–20 mA, 0–10 V, RS485, SSI (per series; options confirmed at order).
- LVDT displacement sensors: ±0.5 mm to ±500 mm, ±0.1% FS linearity, ±0.01% FS repeatability — see LVDT sensors and LVDT vs magnetostrictive.
Download the SGM-PR10 / SGM-PE10 datasheet (PDF)
Non-linearity, repeatability, resolution, outputs, dimensions and ordering codes. No registration required.
Need to check a precision budget for your machine?
Send the stroke, required accuracy or repeatability, output type, controller input and your annual volume. Our engineers will propose a model and show the figures in micrometres. Qualification samples 1–5 pcs in 5–7 working days · production lead time and MOQ confirmed with the quotation · reply within 24 hours.
Factory-Direct Position Sensor Supply for OEMs
SEGMEN (Shenzhen) Industrial Development Co., Ltd. has manufactured sensors in Shenzhen since 2019. Magnetostrictive models for hydraulic cylinders include the SGM-PR10 in-cylinder position sensor; for tanks, see the magnetostrictive level transmitter. Quotations are FOB China with 100% T/T in advance, and every RFQ is answered within 24 hours.
Frequently Asked Questions
No. Resolution is the smallest change the output can show; accuracy is how close the reading is to the true position. A magnetostrictive sensor can resolve 0.1 µm and still have a non-linearity of ±50 µm or more over its stroke.
It is a floor. Use the larger of the two: the percentage times the stroke, or the minimum. For ±0.01% FS on a 300 mm stroke the percentage gives 30 µm, so the ±50 µm minimum applies.
For machines that return to taught positions, yes, because a constant offset cancels out. When the reading must match an absolute dimension, accuracy (non-linearity) matters more.
Usually. A 16-bit analog output divides the stroke into 65,536 steps, about 15 µm each on 1000 mm, and the PLC input card can limit it further. SSI or RS485 outputs pass the position as a number without that conversion.
Not exactly. Non-linearity is the largest part of the accuracy budget at reference conditions; temperature effects, zero offset and mounting add to it in the machine.
Qualification samples of 1–5 pcs ship in 5–7 working days. Production lead time and MOQ are confirmed with the quotation, which is FOB China and answered within 24 hours.
Related reading: Magnetostrictive position sensors — product page · How a magnetostrictive position sensor works · LVDT vs magnetostrictive sensor · Accuracy: % FS vs % of reading · 4–20 mA calculator
Last updated: 25 October 2026. Specifications reflect the SEGMENsensor series datasheets at the time of writing; calculated values are marked as such.