March 2008

Last month, we discussed some rules of thumb for working paper documentation and I invited readers to share some of their rules with me.

Sefton Boyars responded with guidance about sample selection and I asked him to share his knowledge with us. Sefton is a retired Regional Inspector General and enjoys traveling to exotic locations to lead audit training seminars. He also constructs crossword puzzles for USA Today. His email address is seftonboyars@yahoo.com. Enjoy Sefton’s article:

Be Careful When Choosing a Sampling Method!

By Sefton Boyars

Too often, we select an audit sample casually, without thinking through the implications of what we will do with the results. The first mistake often occurs before we draw a sample at all.

Internal controls come first!

Before thinking about sampling, we should have assessed internal controls. We should expect that the sample will corroborate our conclusions regarding the area under audit.

In general, if we consider the internal controls as strong, we would draw a sample just to ensure that the procedures were actually implemented and are working as we anticipate. If the sample confirms our original conclusions, little more needs to be done. Either a judgmental or a random sample will usually work in this instance.

On the other hand, if we conclude that internal controls are weak, we will have to draw a larger sample. (This ties in to the model of audit risk. That is, when control risk is high, we need to compensate with more testing.) The chances of problems increase with a poor internal control system.

Fortunately, some organizations have diligent, honest employees who try to do the right thing even though there are gaps in the controls. In those cases, even the poor internal controls will not result in significant deficiencies.

When internal controls are poor, think twice about what type of sample to draw

When the internal controls are poor, we need to think carefully about the kind of sample we draw.

Statistical sampling: A statistical sample can provide a valid, defensible, picture of the entire universe. Unfortunately, that virtue comes at a cost. Such samples tend to be larger than we would like, and the resulting projections may have wide confidence limits. That is, we may not be very sure that the projection is close to the actual number in the universe.

Judgmental sampling: As an alternative to a statistical sample, we can select a judgmental sample. In a judgmental sample, we select cases for a particular reason. Often, we select cases because we believe they may have a higher than normal likelihood of having deficiencies. Sometimes we just select the largest items.

Regardless of the reasons for selection, we can draw no conclusions about the cases we did not examine. If, for example, we selected items we thought were likely to have deficiencies, we could not in fairness project the results from those transactions to items we did not expect to have deficiencies. Similarly, we cannot assume that smaller items will have the same type or proportion of errors as the larger items.

Statistical sampling may not always be best

I required my staff to explain the rationale for their samples. I wanted to know how they would select the items for testing and how they planned to use the results. Generally, my default was a statistical sample. An auditor rarely had a difficult time convincing me that a statistical sample was the best way to go. But that was not universally true.

We know that when we plot accounting data, it rarely forms the classic bell curve. Rather it is skewed significantly to the left. The vast majority of transactions are small in amount, and a few large items often comprise a large percentage of the total amount in the universe. For that reason, it is often advantageous to select a sample of large items.

Imagine, for instance, that we could draw a definitive conclusion about 90% of the total universe by examining just 8% of the transactions (not that far-fetched a possibility). If that were the case, it would be far more efficient to select that sample than review a statistical sample.

Haphazard sampling

The poorest samples were those that auditors called judgmental samples, but could not explain why they picked the items they did. They just “grabbed a few cases.” To me, this was not a judgmental sample. After all, there was no judgment. I referred to them as haphazard samples. They were the worst of all worlds.

We could not draw any overall opinion about the universe. We knew the results of our tests, but we had little knowledge of how the items in our test related to the universe. As a result, our hard work gave us little information on which to draw audit conclusions.

Record your selection method in the audit program

In summary, select those samples with forethought. Know how you will select sample items and how to plan to use the results of your tests. Record those thoughts in your audit program. The program is more likely to meet the audit planning standard, and you will probably have a successful audit.