Every once in a while, people ask about acceptance sampling plans and I get all riled up. We all know (especially in this political season) that humans are addicted to their indignation high, so here’s your fix for today.
Back when defective products and services were considered inevitable, the military created a standard to try to control just how much defective product was allowable before it would be rejected. This is totally antithetical to the modern concept of quality (conformance to target) or Six Sigma (conformance to specification). And yet, hard as it is to believe, companies are still using these plans. So what’s so bad about them, and how can you help a company move away from them?
You can kind of see the original purpose of acceptance plans. If we assume that defective product is inevitable, then we had better figure out a way for making sure we don’t get too many defective products. To this end, the creators of MIL-STD-105 (and its descendants, e.g., ANSI/ASQ Z1.4) used the concept of inferential statistics (based on the binomial distribution) to create a sampling scheme that could be used by a customer to determine if they should accept a shipment from a supplier. They constructed a series of tables and rules that would allow you to determine how large a sample to take, and the number of “bad” samples allowed before you would reject the shipment.
We can already see how this clashes with the modern understanding of quality. For example, if you received a batch of 500 primers for ammunition and you had determined that the average acceptable quality level (AQL) is 1 percent, then you would randomly sample 50 items from the batch and test them. Based on the binomial distribution, if you found one or less primer defective, you would accept the lot, two or more and you would reject it. (Of course, this inspection could be done at the vendor’s location as well.)
I can hear you clearly across time and space: “Waaaait a minute! You mean that even though in inspection I actually found a bad part in a fairly small sample I’m going to accept the lot anyway?”
Yes indeed. And you will do it while knowing that it’s likely that there are more defective parts in there (on average, five per batch). The reason is that these acceptance sampling plans were built to control for “producer’s risk.” Producer’s risk is the probability that a batch will be rejected when in fact it meets the AQL. This protects the producer, but it doesn’t really address the company buying the primers.
That’s the other risk we would need to consider—consumer’s risk. This is the risk that the lot contains more than the AQL even when we pass it. To illustrate this, let me tell you a (somewhat simplified) true story.
I consulted and trained for a really great guy in a high-tech industry—someone who saw how the use of data could save a company money. He sent me an e-mail one day telling me that he had moved to a smaller new company in the same industry, and that he could use some help with an issue.
He wrote:
“My bosses say that I am spending too much money on quality control. We have an AQL of 1 percent by contract, and they keep pointing to the fact that while we are allowed one defective per lot sample, we have a number of lots that have zero defectives in the sample. This indicates to them that our quality is too high, so we should not spend any money on improving quality. In fact, they say we should look at some ways to save money in the manufacture since we aren’t ‘using up’ our allowed defective margin. I feel that our quality is terrible and should be spending more, but I just can’t compete with that argument."
Now, as a quality professional reading that paragraph, I’ll bet you saw more bad assumptions in that paragraph than you do in political speeches of the other party (whichever party that is).
The first flawed premise
One of the many aspects that his bosses were missing was an understanding that while over time the process might average 1-percent defective, there was no guarantee that any particular lot was. For example, let’s say that we have a lot that actually has 7.5 percent defective units, and we sample 50 and use the rejection criterion of one defective. What’s the chance that we will pass this lot as acceptable, even though it has a much higher defective rate than the AQL? The company managers thought it was impossible. As it turns out, we would accept such a lot about 10.25 percent of the time, on average.
One out of 10 batches, folks. They sold many, many batches per year.
Or, less extreme, if we made a lot with twice the defectives as allowed (2 percent) we would accept that lot 73.6 percent of the time. To show this concept graphically, here’s what is called an OC curve, which tells you the probability of accepting a lot with a given defective rate, for this sampling plan.
What about those lots that found zero defectives in the sample—does that mean that the entire lot is free of defects? Not at all. A lot that was actually 4.5-percent defective would pass a sample of 50 with no defectives found about 10 percent of the time.
All we’re doing with this sampling regime is controlling for the producer’s risk of rejecting an “acceptable” lot (1 percent or less defectives), while leaving out of the discussion how to prevent unacceptable lots (more than 1 percent defective) from being accepted by the consumer. There’s a way to somewhat decrease the consumer’s risk by changing the inspection level, but most businesses I have run across use the default levels because they result in smaller inspection sizes.
There’s also a provision in acceptance sampling to increase the sampling rate because you have experienced a worse than expected defective rate. The idea is that if the data indicate that you are producing above the AQL then you should take larger samples to reduce the risk of sending out lots that are higher than the AQL. However, acceptance sampling only works if you follow those rules, and in the case of this business, they found the rules “too complex” and regardless of lots being rejected, so all bets were off about what the actual rate might have been.
As you can see by looking at the graph, an AQL of 1 percent in no way means that all the lots you send to your customer have a maximum of 1-percent defective. At best, if you follow all the rules and change inspection rates when directed, your average rate might be 1 percent. Of course, this particular process wasn’t producing a stable defective rate anyway—some lots would be fine, others would have high defective rates—calling into question even if an AQL had meaning.
Back to the story: after we tried to convince the managers of this, a major customer received a shipment of 10,000 units that was passed according to the acceptance sampling plan. Sadly, when they began integrating the units into the final product, the product failed 7 percent of the time.
They weren’t amused.
There was little solace in telling them that the company did inspect the lot using the acceptance sampling plan. Their experience now tells them that the product is junk. The managers of the company are shocked, shocked that my contact there could have allowed such a bad shipment go to the customer—didn’t he understand how important this was?
What next?
What’s the second main flawed premise, and how can you move away from acceptance testing to a modern quality system? This was of more than academic interest to this company. I’ll tell you the rest of the story next month, but feel free to discuss this topic by using the link below.
And now that you have had your recommended daily allowance of indignation, go cold turkey for a bit, and, if you are an American citizen, go vote without that monkey on your back. At least until my article next month…
