Measuring the Effectiveness of Checkoff Programs

Evaluation also includes an assessment of whether the organization is "doing the right things;" that is, whether the program goals and objectives and the process designed to meet those goals efficiently and effectively lead to the optimum allocation of the program funds. Even if all promotion expenditures are found to generate positive returns, the evaluation results may suggest some reallocation of funds among alternative activities to maximize the returns to the available funds.

Evaluation also has proven to be important in recent court challenges to checkoff programs. As Crespi and McEowen discuss in another article in this issue of Choices, the constitutionality of many legislatively-mandated commodity programs has been challenged by some who are required to pay as a violation of their First Amendment rights to freedom of speech. Implicit in the arguments is the question that program evaluations are designed to answer whether the promotion and advertising programs funded by checkoff funds are effective in securing the anticipated benefits for those who pay for the programs.

Defining Checkoff Program Effectiveness

Measurement Methods

Promotion → Consumer Awareness → Sales/Prices → Contributor Profits

Consequently, typical approaches to measuring the effectiveness of checkoff programs generally fall into one of three categories: (1) consumer awareness studies, (2) retail sales impact studies, and (3) contributor profit studies. Effectiveness in one category does not necessarily imply effectiveness in other categories. For example, the organization may increase consumer awareness, but not increase either retail sales or profits. By the same token, increases in retail sales may not necessary lead to increases in industry profits.

Consumer Awareness Studies

One problem with these types of studies is that attitudes can be influenced by several factors other than the promotion program so that changes in consumer attitudes/beliefs, as indicated by "tracking studies," cannot always be confidently attributed to the promotion program. For example, even though the 'Other White Meat' message of the U.S. pork industry by itself may have had a positive effect on consumer attitudes about pork, consumer surveys might indicate no change or even a negative change in those attitudes/beliefs if public health messages have simultaneously conveyed concerns about the health risks of eating meat.

Even if the promotion successfully changes attitudes, there is no guarantee that the attitude change will translate into increased sales. As a consequence, many researchers have preferred to analyze the direct relationship between promotion expenditures and sales without considering whether the promotion had any impact on consumer awareness or attitudes.

Retail Sales Impact Studies

Over the years, increasingly sophisticated statistical methods have been developed to isolate and measure the unique contribution of promotion programs to the performance of commodity sales and the profitability of the farm sector. Most common has been the use of econometric models to statistically disentangle the effects of promotion program activities on commodity sales and demand from those of other market forces. Even if the analysis indicates that promotion programs have had a positive and statistically significant effect on market sales, however, the question remains as to whether the increase has been greater than the cost of the program. For that reason, most checkoff organizations are more interested in some measure of return on investment (ROI) rather than the effects of promotion on the level of sales. Consequently, what most studies provide is some Aggregate Measure of the Effectiveness (AME) of checkoff program activities. Unfortunately, the AMEs estimated for checkoff programs often are presented in different forms and calculated in different ways for different commodities, which causes confusion among researchers, as well as among checkoff program managers and stakeholders.

The most commonly reported AME is the benefit-cost ratio (BCR), which is typically calculated in retail sales impact studies as the dollar increase in sales per promotion dollar spent (retail BCR). Because promotion expenditures occur over time and have different effects over their life cycles, the increase in retail sales generated by the program over time often are discounted to present value before dividing by the cost of the program to account for the time value of money. However calculated, if the BCR is greater than one, the promotion program is deemed "effective" because more than one dollar in sales is generated for every dollar spent. On the other hand, if the calculated BCR is less than one, the program is deemed "ineffective."

Because they provide measures of the "average" return to promotion activities, the usefulness of BCRs for making promotion funding allocation decisions is limited. Thus, some studies report a marginal rate of return (MRR) as a more appropriate ROI concept than a BCR as a measure of the advertising and promotion effectiveness. A retail sales MRR is usually calculated as the percentage increase in sales revenues from a 1% increase in promotion expenditures. Thus, an MRR provides a more accurate indication of the change in total returns that might be expected from a reallocation of funds among competing promotion activities.

While BCR and MRR measures provide some sense of whether a checkoff program has effectively increased retail demand and sales, they provide no clear criteria for judging whether the benefits of a particular advertising program have exceeded the costs sufficiently to warrant continuation of the program. How high must a BCR or MRR be in order to justify a conclusion that the program is "effective"? How high is too high and how low is too low?

Reported BCRs for checkoff programs typically range from about 2:1 to 10:1 (Williams & Nichols, 1998; Kaiser et al., 2005). Does that mean that a checkoff program with an estimated BCR of 10:1 is 5 times more effective than a program with a BCR of 2:1? Are BCR estimates of 50:1 or 100:1 unreasonably high or are those programs just that much more effective than programs with more typical BCRs? How are we to interpret a BCR for a checkoff program at the bottom of or below the typical range? Beyond indicating that the cost of a checkoff program is greater than the returns generated, is there any meaningful difference between a BCR between 0 and 1 and a negative BCR?

The problem is that a typical benefit-cost analysis of a checkoff program's effectiveness fails to address whether or not the program is a "good" investment for those who pay for the program. Even if the estimated BCR from a particular advertising program is estimated to be positive and even higher than those estimated for other advertising programs, what program contributors want to know is whether they could do better with the funds they contribute by investing in other common investment opportunities and realizing an even higher return. If so, then it would likely make little difference to them if the BCR from the advertising program is "positive" if they could keep their money and invest it in other investment opportunities and realize a higher return.

For economists, this issue implies that the fundamental concern in measuring the effectiveness of checkoff programs probably should be the opportunity cost of the checkoff program funds from the collective group. This issue has received relatively little attention in the literature. For program managers, the implication is that more relevant information might be provided by economic evaluations if researchers treated checkoff programs as investment alternatives and calculated how well the various programs fare compared to other investment opportunities available … like buying land or investing in Treasury Bills, etc.

The standard business method for determining the highest yielding investment opportunity is to calculate the internal rate of return (IRR) of the investment over time. In analyzing alternative business investments, the IRR often is referred to as the discounted rate of return, the marginal efficiency of capital, and the yield of an investment. For measuring the effectiveness of a commodity promotion program, the IRR is calculated as the change in the future value of the estimated returns to the promotion expenditures over time divided by a change in the present value of advertising expenditures expressed in percentage terms. Consequently, the IRR is a dynamic return on investment measure that expresses the estimated marginal returns to promotion expenditures (i.e., the percent change in returns from a 1% change in promotion).

In a recent study of the Florida orange juice promotion program, the IRR to Florida orange growers was calculated to be 14.4% over the life of the program (Williams, Capps, & Bessler, 2004). In other words, for Florida orange growers to have done better with the funds they invested in the orange juice promotion program, they would have had to have found an investment alternative that yielded more than 14.4% on average annually over the entire 33-year period of the program. Curiously, we tend to use the IRR method for evaluating investments in research that shift the supply curve, but not for investments, like advertising, that shift the demand curve.

Contributor Benefit Studies

Sales impact analyses are designed to determine whether or not past promotion expenditures have effectively shifted out the demand and, therefore, sales. If such analyses conclude that promotion expenditures have not shifted out demand, then it is likely from a statistical perspective that the program has not benefited those who have paid for the program.

However, even if such studies indicate a positive demand impact, the related increase in sales may or may not translate into increased profits of those who pay for the programs for a variety of reasons as discussed by Wohlgenant in another article in this issue of Choices. Most importantly, the benefits of the program may be captured by wholesalers, distributors, processors, importers, foreign producers, or others along the commodity supply chain or even in closely related markets that do not contribute to the costs of the program. For example, in an analysis of the Florida orange juice promotion program, the increase in orange juice demand and price generated by the program prompted an increase in orange juice imports, which benefited foreign orange growers and limited the benefits of the program to Florida orange growers who pay for the program.

Measuring the benefits of promotion programs to those who pay for them requires a more sophisticated and dynamic type of commodity market model than used for demand and sales impact analyses. Because most products pass through several stages of processing before reaching the final consumer, the markets associated with these different stages are interrelated at some level. In vertically related markets, what happens in one market or processing stage affects all other markets or stages. Furthermore, product processing often results in by-products or joint products that sell in entirely different markets. In horizontally related markets, products that are not directly in a processing chain may be considered close substitutes for products in the chain. At the same time, some markets include foreign components. Market supply may include imports and market demand may consist of both domestic and export demand.

The intricacy of the interrelationships among and between markets means that a myriad of factors can affect the transmission of the retail-level effects of checkoff program activities back to those who pay for the program. Once the market for the product has been accurately modeled and the relative roles of the promotion program activities and other market forces at the various levels along the supply chain have been accounted for and incorporated into the model, the process of measuring the benefit of the promotion expenditures to those who pay for the program is done through scenario analysis. This process is accomplished by simulating the model over the historical period with and then without the promotion expenditures included in the model. The actual historical data are taken to represent the "with promotion" scenario. For the "without promotion" scenario, the level of promotion expenditures is first set to zero in the model in each year over the historical period. The model then is simulated over that period to generate changes in the levels of the production, consumption, trade, and prices that would have existed over time in the absence of any promotion program. The simulated differences between the values of model variables in the "with" and "without" promotion scenarios provide direct measures of the historical effects of the program on the market of the commodity being promoted. The results are used to calculate a BCR or an IRR to represent the estimated change in the aggregate profits of the contributors that is attributable to the checkoff promotion program.

Beyond Measurement

Evaluations of checkoff programs specifically are intended to measure the portion of revenues at various levels in the industry that can be directly attributable to the checkoff programs. But in doing so, researchers must compare actual revenues or sales over some time period to nebulous, intangible concepts like "what might have been earned or sold in the absence of the program." In other words, the results imply that $2, $5, or $10 for every dollar they were assessed are in their pockets, but they just don't know it because their earnings would have been lower if the checkoff program had not existed. This concept has proved extremely difficult to communicate to producers.

Compounding that problem is the tendency of many checkoff program staffs and boards to oversell the actual and potential impacts of their programs to insure a positive outcome from producer referenda and otherwise justify continuation of their programs. Contributors come to expect large impacts on prices and profits because of the anecdotes they have been told about how successful various activities have been and how large the benefits to them are from contributing to the program. Estimated BCRs much in excess of 1:1 often are taken to imply large absolute impacts of a checkoff program on the market. Nothing could be less true. A BCR of 5:1 results by dividing a $5 billion industry profit benefit by a $1 billion checkoff investment or by dividing a $5 benefit by a $1 investment. For most commodity promotion programs, the value of the expenditures in research and promotion activities is extremely small in comparison to the total value of industry sales. Commodity promotion expenditures generally amount to a fraction of 1% of the total industry sales each year. With such a low level of investment compared to sales, the overall impact of a commodity promotion program could hardly be expected to be significant in a practical sense in its effects on production, prices, sales, and market share even if the impact could be said to be statistically significant.

When producers and other contributors fail to see the large impact on their returns that they have been led to expect, they tend to disbelieve the measured effectiveness of the checkoff program. One potential solution for checkoff program boards and their staffs is to spend more time educating producers on the true potential of their programs. Perhaps checkoff programs would be better sold to contributors as tools to help reduce downside pressure on prices and profits in bad years and contribute to higher prices and profits in good years rather than as panacea to the financial problems they face.

Another potential solution is to focus on appropriate measures of "effectiveness" that more readily convey the success or failure of checkoff programs in meeting their objectives to program contributors. Benefit-cost ratios as measures of effectiveness have often proved to be less than successful in that effort. Further development of the internal rate of return (IRR) methodology could lead to a measure of effectiveness that might be more easily interpreted by program contributors. Strong arguments also can be made for simple price effects. In a competitive industry, producers relate well to changes in prices as a result of intervention.

Summing Up

Eventually, however, checkoff organizations must determine whether their promotional efforts have gone beyond any change in consumer attitudes to actually shifting out the demand for their commodities. Sales/demand impact studies are designed to measure the retail level impact of checkoff promotion programs. Such studies often report aggregate measures of effectiveness such as benefit-cost ratios or marginal rates of return. Sooner or later, however, someone is going to ask: "What am I getting for my checkoff contribution?" The answer to such questions requires more complex and in-depth analyses to track the retail level impact of advertising and promotion programs back through the supply chain to producers to measure the effectiveness of retail-level promotion programs in enhancing the profitability and economic welfare of producers and other contributors.

Once the effectiveness of the program has been measured, however, the checkoff program still faces the challenge of communicating the results to contributors. Even if the program is deemed to be highly effective, contributors are often skeptical of the magnitude of the results. While they can readily observe the costs of the program to them, the benefits generated are an unobservable component of their total revenue stream. Checkoff program boards and staffs often compound the problem by overselling the potential impacts of their programs on demand, prices, and profits, and by implying that high estimated rates of return imply large program impacts on the market. One solution may be for checkoff program managers to sell their programs as collective efforts to enhance positive market pressures and moderate negative market pressures rather than always shifting out demand and boosting prices. Another solution is for researchers to focus on developing measures that more readily convey the effectiveness of checkoff programs such as the internal rate of return.

For More Information