SC QUALITY GUILDAttribute agreement analysis
SCQUALITYGUILD.COM/TOOLS/ATTRIBUTE-AGREEMENT-ANALYSIS.HTML
Enter how each appraiser rated each part, trial by trial, and the known standard. Get agreement within each appraiser, against the standard and between appraisers, with exact 95% confidence intervals and kappa, plus the parts that cause the trouble.
ON THE EXAM · CSSBB V.C.1, CQE IV.F · FREE · NOTHING YOU TYPE LEAVES YOUR BROWSER
Q-BotClick for an explanation of this tool.For educational use. This page is a study aid. The method behind it has far more depth and detail than one page and one tool can cover, and what you see here is not everything the method offers or everything this site offers. For full understanding and application, use the governing standards and texts, and seek further education, training or qualified advice. Results depend on the data you enter, so check them before relying on them.
What the study is for
A gauge R&R checks a measurement system that gives numbers. Many inspections give a decision instead: pass or fail, go or no-go, a defect grade, a call on whether a weld is acceptable. An attribute agreement analysis is the measurement system analysis for those decisions. It answers two questions. Do the inspectors agree with themselves and with each other? And do they agree with the right answer?
Pick parts that cover the range, with plenty of bad and borderline ones; a study made of obviously good parts makes everyone look perfect. Have an expert, or a measurement, set the standard for each part. Each appraiser then rates every part two or three times, blind, in a different random order each time.
The four assessments
- Within appraiser (repeatability): the share of parts on which an appraiser gave the same rating in every trial, right or wrong.
- Each appraiser vs standard (accuracy): the share of parts on which every one of the appraiser’s trials matched the standard. This is never higher than the within figure.
- Between appraisers (reproducibility): the share of parts on which every appraiser gave the same rating in every trial.
- All appraisers vs standard: the share of parts on which every rating matched the standard. It is the strictest figure and the one that describes the inspection as a whole.
Each percentage is a proportion of a small number of parts, so each has a wide confidence interval. The intervals here are exact binomial (Clopper-Pearson) intervals. With 30 parts, an appraiser who matched on 27 (90%) has an interval of about 73% to 98%: the study cannot tell that appraiser from one who is right three times in four.
Kappa
Two inspectors who both pass 95% of parts will agree most of the time by chance alone. Kappa corrects for that: κ = (Pobserved − Pchance) / (1 − Pchance). A kappa of 1 is perfect agreement, 0 is no better than chance.
- Fleiss’ kappa handles any number of ratings per part. It is used here within each appraiser (the trials are the raters) and between appraisers (every rating is a rater).
- Cohen’s kappa compares two sets of ratings. It is used against the standard, pooling all the appraiser’s trials against the standard repeated once per trial.
A common rule of thumb, given in the AIAG MSA manual: kappa above 0.75 is good to excellent agreement, below 0.40 is poor.
Reading the result
Fix repeatability first. An appraiser who cannot repeat their own decision cannot agree with anyone else. Then look at the table of ratings against the standard. An appraiser who passes bad parts (a miss) is a risk to the customer; one who fails good parts (a false alarm) is a cost. A part rated wrong in every trial points to a misunderstanding of the criterion, which training or a clearer limit sample fixes. Mixed ratings point to a criterion that is hard to judge, which better lighting, a gauge or boundary samples fix.
On the exam
The CSSBB Body of Knowledge lists percent agreement among the measurement system analysis tools and asks for an MSA of attribute measurement systems (V.C.1). The CQE covers measurement system analysis by quantitative and graphical methods (IV.F). Expect to calculate percent agreement from a small table, to say which assessment (within, between or vs standard) a figure describes, and to interpret a kappa value.
Common questions
What is attribute agreement analysis?
The measurement system analysis for pass/fail and category judgments, such as visual inspection or defect classification. It checks whether each appraiser agrees with themselves, with the other appraisers and with the known standard.
What is a good kappa value?
A common rule of thumb reads kappa above 0.75 as good agreement and below 0.40 as poor, with the range between as marginal. Kappa corrects percent agreement for the agreement you would get by chance, so it is the fairer number when one category dominates.
What percent agreement is acceptable?
There is no single standard. Many organizations aim for about 90% or better for each appraiser against the standard and treat lower results as a reason to clarify the criteria or retrain. Look at the confidence interval as well: with few parts, a high percentage can still be uncertain.
How many parts and appraisers should the study use?
Commonly 2 or 3 appraisers, 2 or 3 trials each, and 30 to 50 parts, deliberately including borderline ones, presented in random order without the appraisers seeing earlier answers. Easy parts make any system look good.