Acknowledgments
I. Introduction to Economic Impact Assessements
A. Purpose of the Guidelines
B. What is an Economic Impact Assessment?
C. Why Conduct an Economic Impact Assessment?
C.1. Public Accountability and Documentation of Value C.2. Enhance Research Management
D. Outline of the Guidelines
II. How to Conduct an Economic Impact Assessment
A. Understanding the Stakeholders for the Assessment Study
B. Identify the Population of Candidate Research Projects
B.1. Historical Projects B.2. On-Going Projects B.3. Planned Projects
C. Developing Project Selection Criteria
C.1. Feasibility C.2. Representativeness C.3. Describability
D. Select a Candidate Project
E. Prepare an Historical Overview of the Project
E.1. Chronology of the Project E.2. Technical Outputs from the Project E.3. Current and Expected Users
F. Formulate An Economic Impact Assessment Strategy
F.1. Identify Elements of Cost F.1.a. Direct Costs F.1.b. Indirect Costs F.2. Identify Areas of Benefits
G. Select an Economic Impact Assessment Method
G.1. Guidelines for Selecting an Assessment Method G.2. Alternative Assessment Methods G.2.a. Benefit-Cost Analysis G.2.b. Internal Rate of Return Analysis
H. Conducting the Economic Assessment
H.1. Cost Considerations H.2. Benefit Considerations H.3. Data Collection Process
I. Presentation and Interpretation of the Assessment Findings
A. A Numerical Illustration of Alternative Assessment Methods
The purpose of these guidelines is to familiarize program and project managers at the National Institute of Standards and Technology (NIST) with the motivation for and mechanics of an economic impact assessment. It is intended that this overview information will assist managers:
The purpose of an economic impact assessment of a NIST research project is to evaluate, both qualitatively and quantitatively, the benefits to industry that are associated with the research project and to compare those benefits, in a systematic manner, to the costs of conducting the research project.[1]
An economic impact assessment is different from other evaluation efforts that take place at NIST. For example, laboratory directors regularly conduct or contribute to program evaluations. The purpose of these evaluations is to determine how well the portfolio of research projects within an industry- or technology-focused programmatic area aligns with the objectives of the program or laboratory; to understand how effectively the program is being managed; and to assess progress toward program and broader NIST objectives. Thus, a broad range of evaluation-based metrics are considered, including quantity and quality of technical work, intensity of interactions with constituents, level of satisfaction among constituent groups, variety and quantity of technical output, and some general indication of industry impacts.
In contrast, economic impact assessments focus on the changes in financial and strategic variables within industrial organizations whose activities are directly affected by NIST research and associated services.[2] Impact assessments are therefore narrower in scope than program evaluations and, in fact, contribute to overall program evaluation. Economic impact assessments are generally conducted on completed or on-going projects that have been funded as a result of some prior project selection process. Such assessments are not intended to identify new research areas or to replace standard research project selection exercises, but they frequently contribute to program planning.
There are at least two reasons for the laboratories at NIST to conduct economic impact assessments:
Research at NIST is funded with public resources, and therefore NIST, like any other government agency is accountable for the efficiency with which these resources are used.[3] The recent history of policy initiatives demonstrates the increasing concern about accountability with respect to the efficient use of public resources. The issue of accountability underlies the Competition in Contracting Act of 1984 (P.L. 98-369), the Chief Financial Officers Act of 1990 (P.L. 101-576), and most recently the Government Performance and Results Act of 1993 (P.L. 103-62).[4]
Also, project managers will understandably become advocates for their own research. Watching results on a daily basis and witnessing technical accomplishments understandably leads these managers, and the researchers working under them, to the intuitive conclusion that the project is valuable. Regardless of the veracity of this conclusion, such perceptions may not be easily or convincingly communicated to others. Moreover, in-house criteria for assessing effectiveness are often different from those of policy makers who have, so to speak, the final say on a project's effectiveness.
Thus, when policy analysts or budget analysts ask, "But how do you know this project is effective?" project managers often find themselves relying on "activity" indicators, listing technical accomplishments, or simply telling anecdotal success stories.[5] Although each of these categories of information has some utility, none completely or accurately captures the scope and magnitude of economic impacts the bottom-line metric for policy makers.
It is possible through a systematic economic impact assessment to document value and thereby provide a clear, more precise response to the question of accountability. Future NIST budgets will increasingly depend on the demonstration of economic impacts.
The results of economic impact assessments provide information that can be used to improve both overall managerial efficiency and strategic planning. The findings from such assessments can identify the research outcomes that are most useful to the program's constituents (e.g., industry), as well as define any barriers that might exist which inhibit fuller access to these research results. With such information, management will increase the credibility of its forecast impacts, which is not only critical to strategic planning but also fundamental to the budget process.
At the same time, the information gained from an assessment provides insights into the types of companies, stages in the technology life cycle (e.g., commercialization, fast growth, maturity), or states of economic activity (e.g., R&D, production, marketing) that benefit the most from NIST research. Such information can be an important contributor to strategic planning.
The following section outlines, in a step-by-step fashion, the cognitive and investigatory steps to follow when conducting an economic impact assessment. The text is written from the perspective of a consultant, and the issues that concern a consultant therefore motivate much of the format and content. The reason for this approach is to help NIST technical managers better understand their requirements when participating in an assessment, and to help select a competent consultant.
This mode of presentation is not intended to suggest that economic impact assessments are conducted in a cookbook-like fashion. In fact, no template exists that applies to all project assessments. On the contrary, economic impact assessment is partly art and partly science. The art of assessment is in the creative tailoring of available techniques and methods to the specific project at hand and to the available data. While there are many standard conventions in assessment, such studies are seldom routine. The practice of economic impact assessment involves a series of judgments in the application of assessment methods and the creation of research designs that are only partly standardized. Creative techniques are frequently required in the actual collection of data in order to attain acceptable response rates. The science of assessment is focused in two critical areas: the design of the survey instrument and the various techniques and methods developed for analyzing data and presenting results.
There is an appendix to the guidelines that numerically illustrates two alternative assessment methods.
The first question that a consultant will ask is, "Why is the assessment study being conducted?" The purpose of this question is not to attempt to bias the results of the assessment in any way, or to purposely focus it to highlight a program manager, project manager, or researcher. Rather, it is important to have an up-front understanding of the purpose or need for the study in order to provide an initial indication of what types of impacts were intended when the project was formulated and what types of information are important to the audience(s) for the final report.
One group of stakeholders is the project's managers. Objectives in this case will include a better understanding of how the project being studied aligns with the broader goals of a program or how the types and magnitudes of the resulting economic impacts compare with other projects. They may find this information useful for the general management of the project and the broader set of projects in which they are involved. Additionally, they may have been asked by others to conduct the study as part of a higher level review, which implies emphasis on the project's contribution to broader program objectives. Other groups of stakeholders include policy makers in the executive branch who oversee technology infrastructure policy such as the several management levels above NIST in the Department of Commerce and in the White House, with the latter including the Office of Management and Budget (OMB), Office of Science and Technology Policy (OSTP), Council of Economic Advisors (CEA), and the National Economic Council (NEC) and policy makers in Congress who oversee the budget process: both the authorization of roles and the subsequent appropriation of funds.
Identifying stakeholders often helps the consultant understand who is likely to read the final report. It is often the case that a project researcher who reads an assessment study will look primarily at the technical details in the report and will make a judgment about its validity and usefulness on that basis; a project manager may be looking primarily for information to help allocate resources more effectively; a laboratory director may look primarily for bottom-line information that can be used to summarize to others the net value of the projects to its consumers of the research (e.g., industry); the NIST Director may look primarily for summary statistics, and anecdotes to substantiate them, so that generalizations can be made and substantiated to administrators in the Department of Commerce or in Congress; and policy makers outside of NIST may look primarily for a well documented and readable account of how resources are being allocated and what economic benefits are being realized. Obviously, the technical detail emphasized in the final assessment report will vary depending on the primary audience.
The above statements are generalizations designed to emphasize that different individuals will read an assessment report and will look for different information. For this reason, the consultant will require a clear understanding of who the stakeholders of the study are and how they might use the results of the study.
In addition to understanding the purpose and need for the assessment study, it is imperative for the consultant and the stakeholders to reach an up-front agreement on the level of resources to be devoted to and the time frame for the conduct of the study. Part of the cost, of course, is the payment to the consultant. However, in-house resources (mostly staff time) must also be expensed to assist with the assessment. The consultant will rely on those involved in the project under study to participate in the assessment process at several levels and at specific times during the conduct of the study. For purposes of discussion, the in-house individual(s) will be referred to throughout the guidelines as the project liaison.
The project liaison will be asked to:
The technical outputs from a research project are discussed in detail in Section E.2 below. These outputs must be well-defined and bounded, and directly related to the NIST research. Defining technical outputs too broadly, such as in terms of basic or fundamental knowledge, and failing to link them to economic impacts, will seriously inhibit the usefulness of an assessment. Basic or fundamental knowledge generated from NIST projects often results in specific and identifiable technical outputs, which in turn can be shown to have been embodied in databases, test methods, calibration services, production process models, standards, and the like. These NIST-generated technical outputs have economic impacts outside of NIST.
Also, as discussed in Section F.2 below, identifying the users of the project's technical output and individuals who can act as "gatekeepers" to sources of economic impact information is imperative for the assessment to be successfully completed. For the consultant to be able to interact with such identified individuals in a meaningful way, the project liaison will, in all likelihood, have to "open doors" by making initial introductions. These activities are perhaps the most critical aspect of involvement by the project liaison, and they may be time consuming.
As one example, when assessing the economic impacts associated with the Electronics and Electrical Engineering Laboratory's power and energy calibration services, the project liaison identified by name the key individuals to contact in each company in the affected industries.[6] Had this not been the case, significant additional effort would have been expensed in locating such individuals within each standard meter and watthour revenue meter manufacturing company to solicit their participation; in fact, some of these individuals might never have been identified.
Regarding the time frame of the assessment study, it is commonplace for project managers to underestimate the up-front effort that they and the project liaison must devote to the study. It is often more realistic to set a completion date for the final assessment after this initial phase has been completed. Past experiences with the evaluation of NIST projects leads to an estimate of one work week of effort from the project liaison throughout this initial phase. Given that the project liaison is key to other research projects, it is often the case in practice that this work-week of effort becomes spread over a several month period. In such situations, the consultant may have to acquire the necessary data directly, which adds to the project's cost and delays completion. To ensure the availability of the project liason's time, higher-level management must remain visibly committed to the project's objectives and timely completion.
While the discussion in the previous section presumes that a research project has been identified, that identification process entails some important considerations. Having some idea of the level of effort that a project manager and project liaison will have to devote to the assessment is relevant for the selection of a candidate project.
The population of "assessable" research projects can be broadly grouped into three categories:
Historical projects are those that have been completed and therefore for which the economic outputs and costs are at least theoretically obtainable. On-going projects, as the name implies, are active and thus there will be future outputs and costs that are unknown, but perhaps can be estimated through extrapolation or other forecasting techniques. Finally, there are planned projects, which may or may not have an estimated budget but certainly have no documentable results. An economic impact assessment can be done on any one of these three types of projects, but there are both advantages and disadvantages with each that should be considered.
The advantage of selecting an historical project to study is that all outputs and costs are theoretically known and identifiable. Because the project is completed, users of the outputs from the project will likely be able to discuss their importance.
A disadvantage of selecting an historical project is that some institutional memory and documentation may have been lost. Moreover, the project may no longer be representative of the type of research project being undertaken in a program (in the event that someone tries to extrapolate the findings from one project assessment to other program projects).
On-going projects can either be long lived or recently begun. Studying an on-going project can mitigate the potential disadvantage of an historical project, namely the loss of some impact data, but it has a potential disadvantage that future outputs and costs are unknown. However, depending on the nature of the project, these future outputs and costs can often be estimated with some degree of reliability. In some cases, an "on-going" project may be an extension of a long-lived project, which means that data collection can suffer for the same reasons found in historical projects. In these situations, it may be possible to divide a long-lived project into subprojects with bounded lifetimes.
Ideally, an economic assessment is designed and implemented at the same time as the research project, and impact data are collected throughout the life of the project. As real-time data collection is the most efficient approach, this is the preferred strategy. However, availability of funds and priorities assigned to completed research projects often preclude this approach. Moreover, NIST laboratory research projects are numerous and typically last at least several years, which makes the a priori selection of projects to be studied difficult. That is, given the relatively large number of projects in a programmatic area, all of which may last, say, five to seven years, a priori selection for real-time data collection is problematic at best. To partially address this problem, one of the purposes of these guidelines is to increase the NIST manager's sensitivity to the need to recognize and note sources of impact data that arise during the conduct of a research project. Such a commitment will significantly improve the quality of impact studies and increase the number that can be undertaken.
Planned projects fall into the category of strategic planning and therefore should be viewed somewhat differently from retrospective or on-going projects. In fact, several completed projects should be assessed in a program area before attempting to forecast the impacts of planned projects, as the former can provide an empirical basis for choosing values for variables in the latter forecasting exercise.[7]
The criteria that a consultant will emphasize when selecting a candidate project are not independent of the purpose of the assessment. There are at least three relevant criteria:
This criterion is perhaps the most important to consider. First, as implied in Section B above, relevant information about the project must be available. This in turn requires that the population of consumers of the outputs from the project must not only be identifiable, but also accessible in a timely manner.
Second, the scope of the user population has to be considered. Some NIST projects affect a large number of industries. This can significantly increase the resources required to complete an assessment. Moreover, some database development or basic standards work can affect multiple industries, and do so indirectly through yet additional industries or secondary standards sources. Such infrastructure research can have large aggregate economic impacts, but are also very difficult to assess. For one thing, several distinctively different user populations have to be defined and surveyed. Moreover, the value added by secondary and tertiary user groups has to be netted out in order to obtain estimates of NIST impacts.
It is also helpful if the candidate NIST project has an expected impact that is reasonably well-defined with respect to industry's own investments that are typically occurring simultaneously with NIST research. This is relatively easy to accomplish when the infrastructure contribution is targeted at a single stage (e.g., production) in a single industry (e.g., optical fibers). It is more difficult when multiple groups are involved, some of whom are in the public sector (e.g., users of power and energy calibration services) where non-investment impacts such as "administrative efficiencies" are found.
If the results from the economic impact assessment are to be used to generalize to other projects within a program, then the candidate project should be representative of the projects that are currently underway or expected to be conducted within that program.[8] Dimensions of representativeness include:
Given a subset of feasible and representative projects, consideration should be given to the audience for the assessment report. If it is expected that the report will have a high degree of visibility because of the Director's agenda, or because of the political atmosphere in Congress or in other government agencies, a project's describability should be considered. In other words, is the candidate project one that can be described to a wide audience of non-technical individuals in a way that will be easily understood and for which the importance of such research will easily be recognized?
For example, a broad multidisciplinary audience should more easily understand an optical fiber infrastructure project than one dealing with, say electromigration.[10] The physical representation of a fiber and associated measurement problems such as core diameter, cocentricity, and rate of signal loss are generally easier to visualize and understand than the concept of measuring the rate at which metallic atoms migrate from a metal conductor into a surrounding insulator substrate on a semiconductor chip.
Typically, the above criteria will receive different weights depending on a number of factors, such as the number and type of impact studies already completed and the diversity of projects within an organizational unit. Again, depending on the use of the economic impact assessment, various individuals at different administrative levels may be involved in the final selection of a project for study. Once the candidate project is selected, the overall selection process should be documented, and this documentation should be reported or summarized in the assessment report. It is generally advisable to have the consultant participate in writing this.
At this stage, the consultant will work closely with the project liaison to prepare an historical overview of the project. This overview, along with documentation of the project selection process, will become an important part of the final assessment report.
This overview should contain background information on the history of the project, including the:
Describing the technical outputs from a research project is often an easy process for a scientist or engineer, as would be expected given that such an individual is closely involved in the project and is trained to view technical knowledge per se as the direct and meaningful output. However, the consultant, who is likely not to be a scientist or engineer, will emphasize that an easily understood description of technical outputs is needed for general audiences of economic impact assessment, and in some cases, for the conduct of the actual assessment. Such specific categories of technical outputs might include:
Most non-technical readers of an assessment report will be able to identify with such broad categories of research output in a meaningful way.
However, the actual impact assessment will usually require a focus which is at least one level lower in terms of subcategories of technical output. Both the government researcher and the industry users of the research project's results should be comfortable with this level of detail and complexity and must work cooperatively with the consultant to match technical outputs with economic impacts and benefits (as described in Section F.2 below). Once the match between research output and economic impact is achieved, the consultant will aggregate the results to a more general level in the final report.
For example, when assessing the economic impacts of NIST-supported standards by the Electronics and Electrical Engineering Laboratory for the optical fiber industry, it was not sufficient to define the setting of standards as the relevant technical output. Rather, each of the twenty-two EIA fiberoptic test procedures influenced by NIST between 1981 and 1992 (the year in which the study was conducted) was matched with a performance characteristic of optical fiber.[11] And, industry experts were asked to allocate total realized economic benefits among these standards. Similarly, when evaluating the economic impacts of NIST research by the Electronics and Electrical Engineering Laboratory in electromigration on the U.S. semiconductor industry, direct cost-saving benefits to domestic producers of semiconductors as well as indirect benefits to producers of testing systems were explored.[12]
The historical overview of the project should also include a taxonomy of the categories of users of the project's outputs and the names of accessible individuals within each category. Such categories of current and expected users should include:
And, in addition, the project liaison should help the consultant identify co-producers of project's output. For example, it may be the case that other organizations outside of NIST (e.g., other federal laboratories) are engaged in similar research. This is important information when the consultant begins to quantify the benefits associated with NIST's research.
These three groups represent first-level users, although industrial users are of particular importance. Consumers of goods and services provided by these users can also benefit from the project's output, as discussed below in Section F.2.
It is advisable that this historical overview be distributed to appropriate individuals within NIST for comments. Soliciting internal input at this stage not only has the advantage of assuring a priori accuracy in terms of documentable information, but also provides the opportunity for others to offer observations or constructive comments on the relevancy of the categories of users of the project's output, both within NIST and externally. This is especially important if the project is on-going, and prospective uses of the project's results are what will be documented by the consultant.
At this stage, the consultant will begin to identify areas of economic benefits and associated elements of costs as part of the formulation of an economic impact assessment strategy. This stage is critical in the assessment process because the decisions made by the consultant at this point, which will have been greatly influenced by the background information provided by the project liaison, will determine the research methodology used and the scope of qualitative and quantitative information collected.
The tangible (i.e., measurable) costs considered in an
economic impact assessment include direct costs and indirect
costs. Generally, direct costs are those incurred by NIST and
indirect costs are those incurred by industry to implement the
research outputs. In other words, direct costs are public costs
and indirect costs are private costs. It is advisable for all
costs to be accounted for on a cash basis. That is, relevant to
an impact assessment are costs actually incurred, as opposed to
costs encumbered in the current time period but not spent until a
later time period.
The elements of cost that are of concern to the consultant, and hopefully have been collected by the project liaison as part of the historical overview, include all NIST resources devoted to the research project being studied, by year. These annual costs include R&D, equipment, facilities, maintenance, and fully-burdened labor.[13] It is the case that some research projects use other agency (OA) resources. These are relevant costs and need to be included in the assessment, but it is useful to separate NIST costs from OA costs.
One class of outputs from some research projects are services for which fees are charged. Depending on the nature of these services, the consultant may or may not include the NIST expenses to provide the for-fee service as an element of the project's direct costs. If not considered, the consultant is implicitly assuming that the price paid by the industrial users for this service fully represents the benefits they receive (which may or may not be the case). Or if considered, then the consultant will include the services paid for in the calculation of industrial benefits. In either case, NIST's expenses to establish the facility in order to provide the for-fee services must be considered. Often, a portion of collected fees is redistributed back to the project or to other NIST research programs, thus this reallocation is included in next generation research expenditures. Accordingly, this reallocation portion should be accounted for and subtracted from NIST's costs.
Presumably, these direct cost elements have been collected by the project liaison. This was, for example, the situation when assessing the power and energy calibration project within the Electronics and Electrical Engineering Laboratory and the conformance test program for the Structured Query Language (SQL) database language within the Computer Systems Laboratory. Regarding the power and energy calibration project, its annual operating budget in 1994 was approximately $230,000, which included about $150,000 in fees received from calibrating electric power and energy measurement instruments for industry. Thus, only $80,000 was considered to be the relevant annual NIST cost (against which to compare annual benefits).[14] Similarly, a portion of the cost data collected when evaluating standards for SQL test suites was offset by annual revenues from the sale of these test suites.[15]
There are two important categories of indirect costs: so-called "push" costs and "pull" costs.
In some research projects costs are incurred when transferring (or "pushing") the outputs from the research project to the industrial sector. For example, dissemination costs may be incurred in producing and distributing technical findings (e.g., at conferences), or in operating a more formal technology transfer program through a consortium, standard committee, or other mechanism. When applicable, these cost estimates will be needed by the consultant.
It may also be the case that users of the project's outputs incur costs to identify, acquire, and implement (or "pull in") the outputs from the project. When applicable, these costs estimates will have to be netted from the benefit estimates to the users. For example, when assessing the Manufacturing Engineering Laboratory's research in Real-Time Control Systems Architecture, it was learned that the users of the technology devoted substantial resources over a six-year period to adapting the architecture for their specific purposes.[16] Only then were economic benefits realized.
Having identified in the historical overview the current or expected technical outputs from the research project and the users of these outputs, the consultant will begin to estimate the economic impact that these technical outputs are likely to have on industry. To begin, the consultant will focus on broad-based benefits affecting many segments of industry. It is important to have an understanding of these benefit areas prior to gathering data so that the consultant is fully informed. Such general, yet quantifiable, categories of economic impact that the consultant will consider include:[17]
The technical outputs produced through NIST's research have the potential to create economic benefits by:
For example, when studying the Computer Engineering Laboratory's role in establishing Integrated Service Digital Network (ISDN) conformance testing methodologies and standards-based implementation agreements, it was learned that NIST activities allowed users of ISDN communication technology to realize a 20 percent annual increase in productivity through reduced communication costs.[18]
For example, when studying the Physics Laboratory's operation and maintenance of the Facility for Automated Spectral Calibration, also known as the FASCAL laboratory, qualitative information was collected from manufacturers of measurement equipment, lighting equipment, and photographic equipment documenting that, in the absence of NIST's spectral irradiance standards, equipment customers would be forced to accept greater uncertainty in products.[19]
For example, when studying the Electronics and Electrical Engineering Laboratory's research program in electromigration characterization, domestic producers of semiconductors reported time saved in R&D and improved R&D efficiency from implementing the results of NIST's electromigration research, and they valued these benefits to industry at (in 1991) about $4.1 million.[20]
For example, when studying the Manufacturing Engineering Laboratory's feasibility demonstrations of software error compensation techniques, coordinate measuring machine (CMM) manufacturers reported a reduced time to market of between two and five years for CMMs. This time-to-market savings translated into increased manufacturing productivity valued at approximately $79 million over the 1985-1995 period.[21]
For example, when studying the Computer Systems Laboratory's role in establishing conformance testing programs for the SQL database language, SQL producers reported an average of 2.5 work-years per year reduction in effort to resolve procurement disputes with customers.[22]
Because the project liaison will assist the consultant to "open doors" with users of the technical research outputs, the project liaison must have an understanding of the breadth of economic benefits that the consultant will be quantifying. The project liaison must learn to think about NIST's research in such economic terms in order to help communicate the study's results.
Benefit categories can be thought of in terms of levels. For example, the benefit categories in the examples above are referred to as "first-level" benefits, meaning that they result from the direct industrial use of the project's output. Secondary and tertiary levels of benefits are equally as important to understand, and quantify if possible. Second-level benefits are those received by the users of the goods and services provided by the first-level users of the technological infrastructure, and so on to the third-level. Most assessment studies attempt to characterize as many levels as possible, although for quantitative purposes the consultant will set boundaries on the benefit areas considered; that is, in practice they rarely go beyond the first level partly because of the cost of enlarging the study, but more important, because of the difficulty frequently encountered in isolating the project's partial contribution to the economy beyond the first level of impact.[23]
The consultant will select an assessment method to use in the study prior to formulating a methodology for collecting data. It is important that the project liaison understand the consultant's thinking about alternative assessment methods because the project liaison may be in a position at a future point in time of having to answer questions from third parties about the study.
The consultant will select an assessment method that meets a number of accepted evaluation guidelines. Specifically, the consultant will take into account a number of criteria, including:
The evaluation method selected should be appropriate, meaning that it is not only accepted within the evaluation community but also suitable to meet the objectives, time period and nature of the project, and resource constraints. It should be replicable, meaning that at a future point in time the assessment can be updated by another consultant so that the findings from the original study are comparable to those from the updated study. It should be ends-focused, meaning that the method clearly distinguishes between the resulting impact of the research project's output on the users of the research and the means by which NIST generated that output. And the assessment method should be based on measurement metrics that facilitate comparisons across research projects, including research projects in other organizations.
The two most common economic impact assessment methods are
benefit-cost analysis and internal rate of return analysis. As
discussed below, these methods are related to each other. In
some instances, the consultant will perform both. Appendix A.1
to these guidelines numerically illustrates each method in
detail; however, an overview of the economic interpretation of
each method is provided below.
As the name implies, benefit-cost analysis is a method of comparing the benefits from a project to the costs incurred to conduct the project (i.e., the costs to achieve the benefits). For an economic impact assessment, the benefits are calculated as the dollar value of benefits received by the users of the research project's outputs. The costs are calculated as the sum of direct and indirect costs, as discussed above.[24]
Because benefits and costs occur at different points in time, with costs often occurring before any benefits are realized, benefits should be compared to costs in present value terms.[25]
The end result from a benefit-cost analysis is a unitless
numerical value of the benefit-to-cost ratio. If the ratio is
greater than one, then the present value of benefits is greater
than the present value of costs. The consultant will urge the
project liaison not to simply be content with a ratio greater
than one. The ratio itself is often useful to external
stakeholders, but an understanding of the benefit areas that can
potentially be served is fundamental to enhancing overall
research management.
The internal rate of return (IRR) is a generally accepted metric used in evaluation studies. By definition, it is the discount rate that corresponds to a benefit-to-cost ratio of unity.[26]
Other names that have been used in the literature for the discount rate are the "social" or "spillover" rate of return.[27] As such, this measure is frequently used as an index of the returns to society from an investment made with public funds. The higher the estimated internal rate of return, the greater the estimated net benefits to society.
After taking into account all cost and benefit measurement considerations, the consultant will gather the relevant economic data, analyze them, and summarize the findings in a final report.
On the costs side, the consultant will scrutinize the direct costs provided by the project liaison. This scrutiny is not intended to cast doubt on the due-diligence of the project liaison, but rather to ensure to the satisfaction of the consultant that all NIST costs have been included, and that they are defined and collected according to the same criteria used in other impact assessments. Often, if the project liaison has relied on a budget office for the cost figures, the budget office may not have, for example, taken into account the costs associated with generating fundamental principles that feed directly into the candidate project, or they may not have burdened labor costs to account fully for overhead expenses, or they may not have realized that the project incurred "push" costs to transfer the project's outputs to industry. The consultant may also contact several representatives in selected industries to ascertain these costs. The consultant is not the expert on NIST's budget reporting process, but the consultant is an expert on knowing the categories of direct costs to include in an assessment study.
On the benefit side, the consultant will ensure that all potential areas of economic benefits have been identified in the project overview. The project liaison may assist in this step and involve others in the relevant group. The consultant may request a number of meetings with all those associated with the candidate project in order to ensure that these benefit areas are well understood. The consultant will next ask for the names of a few representative individuals in the relevant industries to interview by telephone. The purpose of these informational interviews is four-fold:
On the basis of this information, the consultant will develop a survey instrument to document both qualitatively and quantitatively the economic benefits associated with the project's outputs. The purpose of such an instrument should not vary significantly across assessment studies, but the format and manner in which questions are asked will vary across projects.[28]
The consultant will rely upon the expertise of the project liaison to identify by name and address the population of industrial users of the project's outputs. The population may be derived from NIST contact files, conference or workshop participation lists, membership roles in trade associations or professional societies, etc.
This is a critical stage in the assessment study and one that may be time consuming for the project liaison. Given this population, the consultant, with assistance from the project liaison, will identify a representative sample of individuals to contact. The consultant will describe the purpose of the study and urge industry to complete the survey instrument, but the project liaison may also be called on to encourage timely participation. This involvement with the survey instrument and with industry will help NIST managers become aware of what constitutes an economic impact. Thus, NIST managers will have a better concept of how to engage in real-time data collection for other studies.
The survey instrument will likely vary across projects. Depending on the technical outputs defined for the project, the survey questions may focus on:
For example, when studying spectral irradiance standards, equipment manufacturers were asked the following questions:[29]
Approximately, how many disputes occur per year with customers regarding the accuracy of your measurement equipment?
In your opinion, is this number less than it would be in the absence of NIST's spectral irradiance standards?
If YES, based on your experience in selling products that are not traceable to a national standard, approximately what would be the number of such disputes per year in the absence of the FASCAL laboratory?
Approximately, how many work days does it take to resolve such a dispute?
Approximately, what is the cost to your company of a fully-burdened work year?
For example, when studying electromigration characterization, semiconductor manufacturers were asked:[30]
What has been the approximate cost savings (time saved, improved R&D efficiency, production cost reduction, etc.) to your company from implementing the results of NIST's research program in electromigration?
In addition to these cost savings, has NIST's overall research program helped your company compete more effectively by influencing R&D objectives, say, by stimulating new product or process ideas? Please explain.
For example, when studying power and energy calibration services, domestic manufacturers of standard meters were asked to respond to the following open-ended question:[31]
I understand from NIST that between 1960 and 1986 uncertainty for the watthour meter was 0.05%, and then it dropped to 0.005% after 1986. Please describe and quantify the benefits that your company and customers have received from this lowering of uncertainty.
The survey instrument may be administered by mail or by telephone, or both. For example, when assessing the economic benefits associated with NIST's support of the Integrated Services Digital Network (ISDN), personal interviews were first conducted at an annual conference in order to obtain general background information and to identify participants in the study; then a mail survey was administered; and finally, follow-up telephone interviews were conducted.[32]
The consultant will prepare a draft report that will be circulated within the relevant NIST units for comments. It is important that the report is clearly written so that a lay-person has a "general" idea of the nature of the project studied and is "comfortable" that the benefit categories considered encompass the potential breadth of the project's impact.
Quite often a consultant is unable to quantify all benefit areas, but will generally be able to assess them qualitatively. Often, such a presentation lends credibility to the overall analysis because most lay-people would be skeptical of a report that purported to quantify all benefit dimensions. If quantitative information is used, the consultant will state that the findings from the study are lower-bound estimates because several categories of benefits have been excluded from the quantitative conclusions. For example, when evaluating several Advanced Technology Program-sponsored projects that were mid-way through their research project, qualitative benefit information was the most appropriate type to gather and report.[33]
The consultant's report will emphasize assessment methods and the associated findings. Such quantitative findings should be interpreted with care. If the consultant emphasizes a benefit-to-cost ratio, that number, in isolation from comparable project assessments, will only tell the reader that the measured benefits from the project are greater or less that the associated costs. If greater, it will tell the reader how much greater.
If the consultant emphasizes an internal rate of return analysis, it is important to realize that this percentage return is not the same as a return on investment estimate. The internal rate of return is compared to a minimum acceptable rate of return. If the estimated internal rate of return exceeds this minimum acceptable level, then ex post it was a worthwhile investment. This interpretation issue is discussed in Appendix A.
The purpose of this appendix is to illustrate, using a set of hypothetical data, the calculations of the two metrics commonly used in economic impact assessments. Specifically, the following discussion compares and contrasts the relationship between a benefit-to-cost ratio and an internal rate of return measure.[34] Because the emphasis of this Appendix is on calculations rather than measurements, it is assumed that:
Consider the data presented in Table A.1 at the end of the Appendix. The project being evaluated began in 1980 and was terminated at the end of 1993. In its first year, $400,000 in capital equipment and research time was allocated to begin the project. Annual costs fell thereafter owing to a decreased need for new capital equipment and an increase in revenues from user fees (e.g., charges for calibration services).[35] At least some companies in the affected industry(ies) reported, through the data collection instrument(s) used by the consultant, that $100,000 in benefits were realized in the first year of the project. Reported benefits increased through 1991, and then decreased through 1994. Although the NIST research project is assumed to have been discontinued in 1993, some companies reported benefits in 1994.
The following formula for the calculation of a benefit-to-cost
ratio was shown and discussed in the text material above:
where the subscript t indexes the time frame over which benefits have been realized and the time over which costs have been incurred. In this hypothetical case, n=14 and m=13. For benefits Bt, t ranges from 0 (corresponding to 1980) to 14 (corresponding to 1994). For costs Ct, t ranges from 0 (corresponding to 1980) to 13 (corresponding to 1993).
The selection of the appropriate discount rate r to use in the calculation of a benefit-to-cost ratio is a disputed conceptual issue.[36] Some argue that the discount rate should approximate the return that could be earned from alternative investments had the research project being evaluated not been undertaken.[37] Others argue that the discount rate should approximate the cost to NIST (or to the government in a more general context) if it had to borrow the resources that were devoted to the project being evaluated. In practice, the consultant doing the evaluation will select a specific discount rate and justify it.
For the illustrative purpose of this Appendix, the discount rate used in this example (and justified by certain literature as a reasonable approximation of the cost of borrowing funds, including administrative costs) is the long-term rate on government bonds plus two percentage points. In this example, a 9 percent discount rate is used (r=0.09).[38]
The benefit-to-cost ratio formula discounts all costs and benefits back to 1980. Stated alternatively, the formula above is the ratio of the discounted present value of benefits and costs. Using the data in Table A.1, the present value of costs, referenced to 1980, is $1,972,550 and the present value of benefits is $8,890,650. Thus, B/C = 4.51. This ratio value should be interpreted to mean that the benefits realized by industry over the period 1980-1994 are 4.51 times greater than the costs incurred by NIST to generate them.
This metric is correctly viewed as a descriptive piece of information. By itself, it cannot be interpreted to mean that this project was successful or unsuccessful because such an interpretation implies that some predetermined value of B/C is optimal. Likewise, one can not infer from this value that the project's funding should have continued beyond 1993. Those are managerial decisions that are correctly determined in light of the laboratory's mission, the results from other projects undertaken concurrently, and the prospects for alternative projects that can be pursued in the future.
An alternative approach, although infrequently used by evaluators, for the calculation of a benefit-to-cost ratio is to inflate (rather than discount) historic costs and benefits to the present by a price index, such as the Consumer Price Index (CPI). Values for the CPI are also shown in Table A.1. The relevant index to divide by each year's costs and benefits is determined by dividing each value of the CPI by 148.2, thus creating an index whose value is 1.0 for 1994. The ratio of the sum of the so-called inflation-adjusted or real benefits to real costs is 5.25. This ratio should be larger than the discounted one because the discount rate used to reference values back to 1980 is larger in value than the rate of inflation used to reference values forward to 1994. Obviously, the usefulness of this approach is that it avoids the issue of selecting a discount rate, but by so doing it ignores the fact that NIST resources do have alternative productive uses.
One could interpret these two benefit-to-cost ratios to mean:
The following formula for the calculation of an internal rate
of return was shown and discussed in the text material above:
NPV = [(B0-C0)/(1+i)0] + [(Bn-Cn)/(1+i)n] = 0where (Bt-Ct) represents net benefits. In 1980 for t=0, net benefits are less than zero, but they are greater than zero in all future years. In 1994, when n=14, net benefits equal total benefits because costs are zero in that year.
By definition, the internal rate of return is the value of the
discount rate i that equates the net present value (NPV) of the
project being evaluated to zero. Based on the data in Table A.1,
the calculated value of i for which NPV=0 is 1.31, implying an
internal rate of return for this project of 131 percent. Before
offering an economic interpretation for this value, it should be
noted that when NPV=0, B/C=1. NPV can be rewritten as:
where, as previously noted, r is the discount rate used to reference future benefits and costs to present value. When NPV=0, then it follows that:
or that the present value of benefits equals the present value of costs, or B/C=1.
An internal rate of return estimate of 131 percent is interpreted to mean that 1.31 is the value of the discount rate that equates the present value of benefits to the present value of costs. As stated in the text, economists and policy makers generally use internal rate of return measures, for on-going or completed public-sector research projects, to estimate what is referred to as the social rate of return. As such, one can infer from the 131 percent value calculated from the hypothetical data in Table A.1 that if 131 percent is above NIST's hurdle rate or generally accepted expected rate of return then the project was worthwhile.[39]
As just stated, the internal rate of return of 131 percent should be compared to NIST's generally accepted rate of return, and if greater, then this project was worthwhile. An alternative way of thinking about the internal rate of return is in terms of a market discount rate. An important behavioral assumption in economics is that the market values future returns from an investment activity by discounting them using a return on an investment of an equivalent risk class. The internal rate of return calculation above can be thought of as providing the discount rate that would result in the market's value of the returns from the research project just equaling the research project's cost, and so the project would break even (i.e., B/C=1). Hence, if the actual discount rate that is appropriate for this research project (i.e., NIST's hurdle rate) is less than 131 percent, then the claim is that the project's returns could be sold in the market for more than the project's costs (i.e., B/C>1).
Some readers might attempt to translate an internal rate of return into an annual yield similar to that earned on, say, a bank deposit. Such a direct comparison is, however, incorrect. The return earned on a bank deposit is a compounded rate of return. One invests, say $100, and then earns interest on that $100 each year plus interest on the interest. That is not the case in an R&D project, in general, or in the hypothetical example shown in Table A.1. Referring to Table A.1, the $400 allocated in 1980 is not reoccurring and the benefits derived from that $400 or from subsequent expenditures are not compounded over time.
Under alternative sets of assumptions, one can calculate implied rates of return from the data in Table A.1, but these rates of return are not the same as the internal rate of return. For example, if all costs are referenced to 1980 using a discount rate of 7.37 percent, total investments in this hypothetical project are $2,098.[40] If all benefits are referenced to 1994 using the same rate, total benefits from this hypothetical project are $26,929. Thus, one could think of this project as one where an initial investment of $2,098 generated, after 14 years, benefits to industry equalling $26,929. Thus, over all 14 years, the total rate of return was 1,184 percent, and the average annual rate of return was 85 percent.
Alternatively, one could calculate the annual compounded rate
of return that corresponds to an initial investment of $2,098 in
1980 growing to $26,929 by the end of 1994.[41] Such a compounded rate of return, x,
equals 20 percent based on the following relationship:
$2,098 (1+x)14=$26,929
One should not expect these two implied rates of return to be identical. Each was calculated under different assumptions. And, if the assumptions change, both calculated rates will change.
While both of these annual percentages have a commonplace
interpretation, the assumptions underlying them do not conform to
the realities of the data in Table A.1, or to the pattern of
investments in research at NIST and the trend in realized
benefits by industry. The internal rate of return is an
accepted evaluation metric and should be interpreted, in
comparison to NIST's hurdle rate, to conclude that a project was
worthwhile or not.
| Year | Costs ($K) | Benefits ($K) |
|---|---|---|
| 1980 | $400 | $100 |
| 1981 | $250 | $600 |
| 1982 | $250 | $600 |
| 1983 | $250 | $700 |
| 1984 | $250 | $950 |
| 1985 | $250 | $950 |
| 1986 | $225 | $950 |
| 1987 | $225 | $1,500 |
| 1988 | $150 | $1,500 |
| 1989 | $150 | $1,500 |
| 1990 | $150 | $2,200 |
| 1991 | $150 | $2,200 |
| 1992 | $125 | $1,500 |
| 1993 | $100 | $1,500 |
| 1994 | $0 | $900 |
Leyden, D.P. and A.N. Link, Government's Role in Innovation, Norwell, MA., Kluwer Academic Publishers, 1992.
Link, A.N., "An Economic Evaluation of the Spectral Irradiance Standard," prepared for the Physics Laboratory at NIST, November 1995a.
Link, A.N., "An Evaluation of the Economic Impacts Associated with the NIST Power and Energy Calibration Services," prepared for the Electronics and Electrical Engineering Laboratory at NIST, January 1995b.
Link, A.N., "Economic Impact on the U.S. Semiconductor Industry of NIST Research in Electromigration," prepared for the Electronics and Electrical Engineering Laboratory at NIST, January 1992a.
Link, A.N., "Economic Impacts of NIST-Supported Standards for the U.S. Optical Fiber Industry: 1981-Present," prepared for the Electronics and Electrical Engineering Laboratory at NIST, February 1992b.
Link, A.N., "Economic Study of the Advanced Manufacturing Technology for Low-Cost Flat Panel Displays Joint Venture After Three Years," prepared for the Advanced Technology Program at NIST, August 1994a.
Link, A.N., "Economic Study of the Joint Venture Project on Short-Wavelength Sources for Optical Recording After Three Years of a Five-Year Research Program," prepared for the Advanced Technology Program at NIST, March 1994b.
Link, A.N., "Economic Study of the Printed Wiring Board Joint Venture After Two Years," prepared for the Advanced Technology Program at NIST, April 1993a.
Link, A.N., Evaluating Public Sector Research and Development, New York: Praeger Publishing Company, forthcoming 1996.
Link, A.N., "Measuring the Economic Impact of the Advanced Technology Program: A Planning Study," prepared for the Advanced Technology Program at NIST, May 1992c.
National Institute of Standards and Technology (NIST), Setting Priorities and Measuring Results at the National Institute of Standards and Technology, U.S. Department of Commerce, January 31, 1994.
Ruegg, R. and H. Marshall, Building Economics: Theory and Practice, New York, Van Norstrand Reinhold, 1990.
TASC, "Software Error Compensation Research in NIST's Manufacturing Engineering Laboratory: An Evaluation of First-Order Economic Impacts," prepared for the Manufacturing Engineering Laboratory at NIST, April 1996.
TASC, "An Economic Evaluation of NIST's Research in Real-Time Control System (RCS) Architecture: A Case Study Approach," prepared for the Manufacturing Engineering Laboratory at NIST, April 1995a.
TASC, "Economic Assessment of the NIST Conformance Test Program for Database Language SQL," prepared for the Computer Engineering Laboratory at NIST, September 1995b.
TASC, "NIST's Support of the Integrated Services Digital Network (ISDN)," prepared for the Computer Engineering Laboratory at NIST, November 1994.
Task Force on Alternative Futures for the Development of Energy National Laboratories, "Alternative Futures for the Department of Energy National Laboratories,: Energy Advisory Board, February 1995.
Tassey, G., Technology Infrastructure and Competitive Position, Norwell, MA, Kluwer Academic Publishers, 1992.
Tassey, G. "Technology and Economic Growth: Implications for Federal Policy," NIST Planning Report 95-3, October 1995b.
Tassey, G., "The Rates of Return to Private and Public
Investment in Technology," NIST mimeo, March 1995a.
[1] Industry is not the only sector that benefits from
NIST's research. Economic impacts accrue to consumers and the
public at large; however, potential users of research outputs
within industrial sectors are of primary importance to NIST.
[2] The specific financial and strategic variables to
focus on are a matter of judgment, as discussed later under the
broader topic of setting boundaries on an economic impact
assessment.
[3] As stated in Setting Priorities and Measuring
Results at the National Institute of Standards and
Technology, "... the need to measure returns on taxpayer
investments in federal technology programs is of paramount
importance ..." (p. 10).
[4] Public accountability was also emphasized in the
1993 National Performance Review report. Therein, the importance
of measuring results rather than inputs was reinforced. In this
context, the Task Force on Alternative Futures for the Department
of Energy National Laboratories reported (in the so-called Galvin
Report) that "... there is a perception that the U.S. government
is spending significant resources on the development of new
technologies but that American industry is not reaping the
rewards of that investment" (p. 46).
[5] Managers are not the only individuals or groups of
individuals that would ask such a question. The Office of
Management and Budget (OMB) and the General Accounting Office
(GAO) conduct evaluations of government agencies like NIST.
Their inquiries across agencies within the government are similar
to the program evaluations across laboratories within NIST. NIST
can more effectively document its overall performance when it has
a portfolio of project assessments to illustrate the
effectiveness of its laboratories' research programs. Also, it
was explicitly noted as one of three motivations for the
enactment of the Government Performance and Results Act of 1993
that "Federal managers are seriously disadvantaged in their
efforts to improve program efficiency and effectiveness, because
of insufficient articulation of program goals and inadequate
information on program performance."
[7] The metrics used for project selection are
different from those used for project assessment.
[8] Of course, the representativeness criterion takes
as given that the assessment is feasible. Because not all
projects are feasible, selection bias may be introduced at this
point. The consultant will address this issue with the project
liaison.
[9] For example. improving the precision of clinical
cholesterol measurements may be more representative of other
chemistry projects which also involve some sort of test method
and the production of a standard reference material (SRM) than,
say, a project for the development of an equation of state for an
alternative refrigerant.
[13] It may be the case that the research project being
studied relies on fundamental science developed elsewhere in
NIST. A portion of the cost to transfer that knowledge to the
project being studied will be considered by the consultant.
[17] These are not independent categories of benefits.
[23] This is a weakness of many of the economic impact
studies conducted at NIST to date. As such the evaluation
metrics associated with these studies are lower-bound estimates
of the true social benefits associated with NIST's research.
[24] It should be noted, as discussed in Link
(forthcoming 1996), that some academics take exception with using
a benefit-to-cost ratio to compare public costs with private
benefits. However, from a project manager's perspective, their
charge is to in fact do just that; namely, to identify the
private sector benefits associated with their publicly-funded
research.
[25] Thus, the ratio of benefits (B) to costs (C) that
is used in a benefit-cost analysis is:
where the subscript t indexes the time frame over which costs have been or will be incurred and the time over which benefits have been or will be realized. Bt represents all measurable benefits in period t, and Ct represents all measurable costs in period t. It is often the case that C&subt;>B&subt; in the early years of a project, and then Bt>Ct in the latter years. If in a future year, t=m for m
[26] Stated alternatively, the IRR is the rate i that
equates the net present value (NPV) of a project to zero:
NPV = [(B0-C0)/(1+i)0] + ... + [(Bn-Cn)/(1+i)n] = 0where (Bt-Ct) represents net benefits. If, as is often the case in the early years of a project, Ct>Bt, then net benefits are negative in that year. Net benefits will be positive when Bt>Ct.
[27] See Tassey (1995a, 1995b).
[28] In fact, there may be more than one survey
instrument, each tailored to a particular group of industrial
users. The program/project manager should be aware that approval
may be needed from the Office of Management and Budget (OMB)
prior to conducting the survey.
[33] See Link (1993a, 1994a, 1994b). The consultant
will make an aggressive effort to quantify all cost elements
associated with the project. The consultant will, however, be
conservative when quantifying benefits. While potentially
frustrating to the project liaison (who may feel that he/she is a
stakeholder for the evaluation study), this tendency to downward
bias the evaluation results adds not only accuracy but also
credibility to the final assessment.
[34] It must be emphasized that the data herein are
hypothetical. A reader should not assume that the growth pattern
underlying either the cost or benefit estimates is necessarily
representative of other NIST research projects.
[35] Thus, the estimates in Table A.1 are net costs.
[36] See Link (forthcoming 1996) for a discussion of
this point and for additional references on the issue.
[37] Economists refer to this concept as opportunity
cost.
[38] NIST may consider establishing its own criteria
for calculating a discount rate to use in impact assessments in
order to ensure some comparability across studies.
[39] There are a number of additional problems
associated with an internal rate of return measure when it is
used for research project selection based on forecasted benefit
data. Because the types of projects that are expected to be
evaluated at NIST are completed or on-going, as opposed to being
proposed, these issues are not relevant here.
[40] This discount rate equals the average yield on
30-year Treasury bonds in 1994, as reported in the Economic
Report of the President, 1995. The year 1994 was selected
because it is the last year for which benefits are measured.
However, there is no methodological reason for the selection of
the terminal year or for the use of the 30-year Treasury bond
yield. This illustration is intended only to illustrate
alternative interpretations for the data in Table A.1 as opposed
to the implementation of an alternative evaluation metric.
[41] This calculated rate of return in similar to the
overall rate of return suggested by Ruegg and Marshall (1990).
