Reference
The calibration reference desk.
A working reference for quality managers, technicians, and machinists: a metrology glossary written by our lab, conversion charts we verified computationally, and straight answers on how calibration levels work. Bookmark it — it beats scrolling a 72-minute page.
Calibration & metrology glossary
59 terms, written by our lab — not scraped.
A
- A2LA
- The American Association for Laboratory Accreditation — an independent, internationally recognized accreditation body that assesses calibration and testing laboratories against ISO/IEC 17025. American Gage is A2LA-accredited under certificate 4296.01.
- Accreditation
- Formal recognition by an independent body that a laboratory is competent to perform specific calibrations. Accreditation covers a defined scope — always check the lab’s scope document, not just the logo.
- Accuracy
- How close a measurement is to the true value of the quantity being measured. Often confused with precision, which describes agreement among repeated measurements rather than closeness to truth.
- Adjustment
- Physically or electronically altering an instrument so its indication matches the reference standard. A calibration measures; an adjustment corrects. As-found data captures the state before adjustment.
- AMS 2750
- The aerospace specification governing pyrometry — temperature sensors, instrumentation, and uniformity surveys for heat-treating equipment. Nadcap heat-treat audits are built around it.
- As-found data
- The readings an instrument produced when it arrived for calibration, before any adjustment or repair. Essential for out-of-tolerance impact assessments and for justifying calibration intervals.
- As-left data
- The readings an instrument produced at the end of calibration, after any adjustment. Together with as-found data, it documents what changed on the bench.
B
- Bias
- A consistent, systematic offset between an instrument’s indication and the true value — as opposed to random scatter. Bias can often be corrected by adjustment; random error cannot.
C
- Calibration
- Comparing an instrument’s indications to a reference standard of known, traceable value under controlled conditions, and documenting the results. Calibration itself does not necessarily include adjustment.
- Calibration certificate
- The documented output of a calibration: instrument identity, standards used and their traceability, environmental conditions, results (ideally as-found/as-left with uncertainty), the decision rule applied, and who performed the work.
- Calibration due date
- The date after which an instrument should not be used for acceptance measurements until recalibrated. Measurements made with an overdue instrument are treated as suspect in most quality systems.
- Calibration interval
- The planned period between calibrations. Standards require intervals to be justified — typically by manufacturer guidance, usage severity, and the instrument’s documented as-found history.
- CMC (Calibration and Measurement Capability)
- The smallest measurement uncertainty a laboratory can achieve for a given parameter and range, as listed on its scope of accreditation. CMCs are what you compare when evaluating labs.
- Coverage factor (k)
- The multiplier applied to combined standard uncertainty to produce expanded uncertainty at a stated confidence level. k=2 corresponds to approximately 95% confidence and is the industry default.
D
- Deadweight tester
- A primary-method standard that generates force or pressure from calibrated masses under gravity. Dead weights minimize inherited sensor error, which is why they deliver the best available uncertainties.
- Decision rule
- The documented method for declaring pass or fail when measurement uncertainty is taken into account — required by ISO/IEC 17025:2017 and central to ANSI/NCSL Z540.3 compliance.
- Drift
- Gradual change in an instrument’s indication over time, independent of use conditions. Drift between calibrations is the core argument for keeping intervals honest.
E
- Error (of measurement)
- The difference between an instrument’s indication and the reference value. Reported on certificates point by point; distinct from uncertainty, which quantifies doubt about the reference itself.
F
- First article inspection (FAI)
- Complete verification of the first part produced by a new or changed process, common in aerospace (AS9102). FAIs depend on inspection instruments with current, traceable calibration.
G
- Gage block
- A precision length standard with lapped, wringable faces, used to calibrate and set dimensional instruments. Working sets are verified against masters; each block should be reported individually with uncertainty.
- Gage R&R
- A study of measurement-system variation split into repeatability (same operator) and reproducibility (between operators). Calibration controls instrument error; gage R&R evaluates the whole measurement process.
- Guard banding
- Tightening acceptance limits by some or all of the measurement uncertainty to reduce the risk of falsely accepting an out-of-tolerance instrument. A common way to implement a decision rule.
H
- Hysteresis
- When an instrument reads differently approaching the same value from above versus below — common in pressure gauges and force instruments. Good procedures exercise the instrument in both directions.
I
- ILAC MRA
- The mutual recognition arrangement among international accreditation bodies. Because A2LA is a signatory, accredited certificates from American Gage are recognized by auditors worldwide.
- In-tolerance
- A calibration result where the instrument’s error at each test point falls within its specified limits, per the applied decision rule.
- ISO 8655
- The international standard for piston-operated volumetric apparatus — pipettes, burettes, dispensers — defining test methods (gravimetric and photometric), test volumes, and error limits.
- ISO/IEC 17025
- The international standard for the competence of calibration and testing laboratories, covering technical methods, traceability, uncertainty, impartiality, and quality management. Accreditation to 17025 is the accepted proof a calibration supplier is competent.
L
- Linearity
- How consistently an instrument’s error behaves across its range. A balance may be perfect at mid-range and out at full capacity — which is why calibrations test multiple points.
M
- Measurand
- The specific quantity being measured — not the instrument, but the thing: the DC voltage at 10 V, the torque at 50 lbf·ft, the mass of the 1 kg weight.
- Metrology
- The science of measurement, covering units, standards, methods, and uncertainty. Calibration is applied metrology.
N
- NIST
- The National Institute of Standards and Technology — the U.S. national metrology institute that maintains the national measurement standards to which calibrations in the United States are traced.
- Nominal value
- The stated or intended value of a standard or artifact — the “1.0000 in” engraved on a gage block. Calibration reports the deviation from nominal with uncertainty.
- Nonconformance report (NCR)
- The quality record raised when equipment or product fails requirements — including when an instrument is found out of tolerance and its measurement history needs assessment.
O
- Out-of-tolerance (OOT)
- A calibration result where the instrument exceeded its error limits. Standards require assessing whether measurements made since the last good calibration were affected, and acting on any impacted product.
P
- Precision
- The agreement among repeated measurements — tight scatter. An instrument can be precise and still inaccurate if every reading shares the same bias.
- Primary standard
- A standard realized from fundamental definitions or primary methods (fixed-point cells, deadweight force) rather than calibrated against another artifact of the same kind. Sits at the top of the traceability chain below national standards.
R
- Range
- The span of values an instrument is designed to measure. Calibrations test points distributed across the range — and “limited calibration” restricts certification to part of it.
- Reference standard
- The highest-accuracy standard a laboratory holds for a given quantity, used to calibrate its working standards. Reference standards are themselves calibrated up the traceability chain.
- Repeatability
- Variation in results when the same operator measures the same item with the same instrument under the same conditions in quick succession. The floor of measurement performance.
- Reproducibility
- Variation in results when conditions change — different operators, days, or laboratories. Larger than repeatability, and the honest measure of real-world consistency.
- Resolution
- The smallest change an instrument can display or detect. Resolution limits precision but does not guarantee accuracy — a 6-digit meter can be confidently wrong.
- Reverse traceability
- Identifying every measurement, product, or lot an instrument touched since its last good calibration — the exercise triggered when that instrument is found out of tolerance.
S
- Scope of accreditation
- The official document listing exactly which parameters, ranges, and uncertainties (CMCs) a lab is accredited for. If a measurement isn’t on the scope, it isn’t accredited — full stop.
- Sensitivity
- The ratio of an instrument’s output change to the input change that caused it — how strongly the instrument responds to the measurand.
- Span
- The difference between the upper and lower limits of an instrument’s range. Process instruments are often specified and calibrated as percent-of-span.
- SPRT
- Standard Platinum Resistance Thermometer — the interpolation instrument of the international temperature scale (ITS-90) and the reference thermometer class used in high-accuracy temperature laboratories.
- Stability
- An instrument’s ability to hold its performance over time. Stability history — visible in successive as-found results — is the evidence base for extending calibration intervals.
T
- Tolerance
- The permissible limits of error for an instrument or a manufactured feature. Calibration determines whether the instrument is inside its tolerance; uncertainty determines how sure you can be.
- Traceability
- An unbroken, documented chain of calibrations connecting a measurement to national or international standards, with stated uncertainty at every link. The property that makes a certificate mean something.
- Transfer standard
- A standard used to carry a value from one place or level to another — for example, a precision digital multimeter used to move a voltage value from the reference lab to a working bench.
- Triple point of water
- The condition where water coexists as solid, liquid, and vapor — exactly 0.01 °C by definition. Realized in sealed cells, it anchors practical temperature calibration.
- TUR (Test Uncertainty Ratio)
- The ratio between the tolerance being tested and the uncertainty of the calibration process testing it. A 4:1 TUR is the traditional benchmark; ANSI/NCSL Z540.3 requires managing false-accept risk when it can’t be met.
- Type A / Type B evaluation
- The two ways uncertainty components are quantified: Type A from statistical analysis of repeated measurements, Type B from other knowledge — specifications, certificates, physics.
U
- Uncertainty (of measurement)
- A quantified estimate of the doubt in a measurement result, combining contributions from standards, environment, method, and instrument. Reported on accredited certificates, usually expanded at k=2 (~95% confidence).
- Uncertainty budget
- The itemized accounting of every uncertainty contributor in a measurement — standards, resolution, environment, repeatability — combined into the reported value. Accredited labs maintain one per measurement type.
V
- Verification
- Confirming an instrument meets specified requirements — a pass/fail check against tolerance. Related to but narrower than calibration, which quantifies the instrument’s actual errors.
W
- Working standard
- The standard used for day-to-day calibrations on the bench, periodically calibrated against the laboratory’s reference standards.
Z
- Z540 (ANSI/NCSL)
- The U.S. national standards for calibration programs: Z540-1 (laboratory and program requirements) and Z540.3 (calibration of measuring and test equipment, including decision rules and false-accept risk). Widely flowed down in defense and aerospace contracts.
- Zero error
- An instrument’s indication when the input is genuinely zero. Simple to check, commonly drifting, and the first thing a good procedure verifies.
Calibration service levels compared
Match the level to what your quality system and customers require — the levels mirror the options on our quote form.
| Traceable, no data (Level 1) | Traceable Z540, with data (Level 2) | Accredited ISO/IEC 17025, with data | |
|---|---|---|---|
| NIST-traceable standards | ✓ | ✓ | ✓ |
| Certificate issued | ✓ | ✓ | ✓ |
| As-found / as-left readings | — | ✓ | ✓ |
| Measurement uncertainty reported | — | — | ✓ |
| Issued under A2LA accreditation | — | — | ✓ |
| Typical use | Basic compliance, non-critical tools | General QA programs needing data | AS9100, ISO 13485, FDA & audited programs |
Not sure which level your auditors expect? Our cost guide explains the trade-off — or ask when you request a quote.
Conversion charts
Every factor below was verified computationally against NIST SP 811 before publishing — because a reference chart with math errors is worse than none.
Pressure unit conversions
Factors per NIST SP 811. inH₂O referenced at 4 °C; values rounded to 6 significant figures.
| 1 unit = | psi | kPa | bar | inH₂O | inHg | torr (mmHg) |
|---|---|---|---|---|---|---|
| 1 psi | 1 | 6.89476 | 0.0689476 | 27.6799 | 2.03602 | 51.7149 |
| 1 kPa | 0.145038 | 1 | 0.01 | 4.01474 | 0.295300 | 7.50062 |
| 1 bar | 14.5038 | 100 | 1 | 401.474 | 29.5300 | 750.062 |
| 1 inH₂O | 0.0361273 | 0.249082 | 0.00249082 | 1 | 0.0735559 | 1.86832 |
| 1 inHg | 0.491154 | 3.38639 | 0.0338639 | 13.5951 | 1 | 25.4000 |
| 1 atm | 14.6959 | 101.325 | 1.01325 | 406.793 | 29.9213 | 760 |
Torque & force conversions
Values rounded to 6 significant figures.
| Convert | Multiply by | To get |
|---|---|---|
| lbf·ft | 1.35582 | N·m |
| lbf·in | 0.112985 | N·m |
| ozf·in | 0.00706155 | N·m |
| N·m | 0.737562 | lbf·ft |
| lbf | 4.44822 | N |
| kgf | 9.80665 | N |
| N | 0.224809 | lbf |
Length conversions
The inch is defined as exactly 25.4 mm.
| Convert | Multiply by | To get |
|---|---|---|
| inches | 25.4 (exact) | millimeters |
| millimeters | 0.0393701 | inches |
| feet | 0.3048 (exact) | meters |
| meters | 3.28084 | feet |
| microinches (µin) | 0.0254 (exact) | micrometers (µm) |
| micrometers (µm) | 39.3701 | microinches (µin) |
Temperature conversions & fixed points
| Relationship | Formula / value |
|---|---|
| Celsius → Fahrenheit | °F = (°C × 9/5) + 32 |
| Fahrenheit → Celsius | °C = (°F − 32) × 5/9 |
| Celsius → Kelvin | K = °C + 273.15 |
| Absolute zero | −273.15 °C / −459.67 °F / 0 K |
| Triple point of water | 0.01 °C (exact, by definition) |
| Water boils (1 atm) | ≈100 °C / 212 °F |
SI prefixes (the useful ones)
| Prefix | Symbol | Factor | Example in calibration |
|---|---|---|---|
| giga | G | 10⁹ | GHz — RF signal generators |
| mega | M | 10⁶ | MΩ — insulation resistance |
| kilo | k | 10³ | kPa — pressure |
| milli | m | 10⁻³ | mV — thermocouple outputs |
| micro | µ | 10⁻⁶ | µm, µin — dimensional; µA — current |
| nano | n | 10⁻⁹ | nm — surface finish, optics |
| pico | p | 10⁻¹² | pF — capacitance |
Fraction → decimal → millimeter chart
All 64ths, exact values (1 in = 25.4 mm by definition), decimals to four places.
| Fraction | Inches | mm |
|---|---|---|
| 1/64 | 0.0156 | 0.397 |
| 1/32 | 0.0312 | 0.794 |
| 3/64 | 0.0469 | 1.191 |
| 1/16 | 0.0625 | 1.587 |
| 5/64 | 0.0781 | 1.984 |
| 3/32 | 0.0938 | 2.381 |
| 7/64 | 0.1094 | 2.778 |
| 1/8 | 0.1250 | 3.175 |
| 9/64 | 0.1406 | 3.572 |
| 5/32 | 0.1562 | 3.969 |
| 11/64 | 0.1719 | 4.366 |
| 3/16 | 0.1875 | 4.762 |
| 13/64 | 0.2031 | 5.159 |
| 7/32 | 0.2188 | 5.556 |
| 15/64 | 0.2344 | 5.953 |
| 1/4 | 0.2500 | 6.350 |
| 17/64 | 0.2656 | 6.747 |
| 9/32 | 0.2812 | 7.144 |
| 19/64 | 0.2969 | 7.541 |
| 5/16 | 0.3125 | 7.938 |
| 21/64 | 0.3281 | 8.334 |
| 11/32 | 0.3438 | 8.731 |
| 23/64 | 0.3594 | 9.128 |
| 3/8 | 0.3750 | 9.525 |
| 25/64 | 0.3906 | 9.922 |
| 13/32 | 0.4062 | 10.319 |
| 27/64 | 0.4219 | 10.716 |
| 7/16 | 0.4375 | 11.112 |
| 29/64 | 0.4531 | 11.509 |
| 15/32 | 0.4688 | 11.906 |
| 31/64 | 0.4844 | 12.303 |
| 1/2 | 0.5000 | 12.700 |
| Fraction | Inches | mm |
|---|---|---|
| 33/64 | 0.5156 | 13.097 |
| 17/32 | 0.5312 | 13.494 |
| 35/64 | 0.5469 | 13.891 |
| 9/16 | 0.5625 | 14.287 |
| 37/64 | 0.5781 | 14.684 |
| 19/32 | 0.5938 | 15.081 |
| 39/64 | 0.6094 | 15.478 |
| 5/8 | 0.6250 | 15.875 |
| 41/64 | 0.6406 | 16.272 |
| 21/32 | 0.6562 | 16.669 |
| 43/64 | 0.6719 | 17.066 |
| 11/16 | 0.6875 | 17.462 |
| 45/64 | 0.7031 | 17.859 |
| 23/32 | 0.7188 | 18.256 |
| 47/64 | 0.7344 | 18.653 |
| 3/4 | 0.7500 | 19.050 |
| 49/64 | 0.7656 | 19.447 |
| 25/32 | 0.7812 | 19.844 |
| 51/64 | 0.7969 | 20.241 |
| 13/16 | 0.8125 | 20.637 |
| 53/64 | 0.8281 | 21.034 |
| 27/32 | 0.8438 | 21.431 |
| 55/64 | 0.8594 | 21.828 |
| 7/8 | 0.8750 | 22.225 |
| 57/64 | 0.8906 | 22.622 |
| 29/32 | 0.9062 | 23.019 |
| 59/64 | 0.9219 | 23.416 |
| 15/16 | 0.9375 | 23.812 |
| 61/64 | 0.9531 | 24.209 |
| 31/32 | 0.9688 | 24.606 |
| 63/64 | 0.9844 | 25.003 |
| 1/1 | 1.0000 | 25.400 |
Calibration 101
What calibration actually is
Calibration is a documented comparison: your instrument’s readings against a reference standard whose value is known through an unbroken, NIST-traceable chain. The output is data — how far off the instrument reads at each test point — not just a sticker. Adjustment, if the instrument supports and needs it, is a separate act that calibration makes visible through as-found and as-left readings.
Why it’s worth paying for
Every acceptance decision your instruments make inherits their error. A drifted caliper quietly passes bad parts or rejects good ones; a drifted torque wrench under- or over-tightens every joint it touches. Calibration is the insurance policy on all of it — and when an instrument does turn up out of tolerance, as-found data is what lets you scope the damage instead of recalling everything it ever measured.
How intervals get set
No law says “annually.” Standards require planned, justified intervals — built from manufacturer guidance, how hard the instrument is used, and its own as-found history. Instruments that keep arriving in tolerance earn longer intervals; instruments that drift earn shorter ones. Our calibration frequency guide has typical starting intervals by instrument type.
In-house vs. outsourced
Calibrating in-house means owning reference standards (and calibrating them), writing and maintaining procedures, training technicians, computing uncertainty budgets, and passing audits on all of it — a full metrology operation, sensible at high volume. Most manufacturers are better served sending instruments to an accredited lab and keeping their engineers on product. If you’re weighing it, ask us what your instrument list would actually cost both ways; the honest math usually settles it.
Reference desk open, lab door too.
When the chart answers the question but the instrument still needs calibrating: send your list and get scope, pricing, and turnaround within one to two business days.