164 



The committee benefited from a large number of studies cited, and also many not 

 directly cited. ^ The committee faced the question of how federal dollars should be 

 invested most effectively, and took the view that the federal government should con- 

 sider the various options in an investment "portfolio" of research and development 

 programs for different ends, through different agencies, and done by different per- 

 formers. To a remarkable extent, the evaluation literature fails to address the rel- 

 ative effectiveness of alternative ways to invest. Rather, most evaluations, quite rea- 

 sonably, assess whether a program is funding the kind of work it was set up to do. 

 This is an example of the "silo" effect of programs being funded and evaluated not 

 by comparison to alternatives within R&D but judged only by some other standard. 

 The committee recommends that the different mechanisms be evaluated relative to 

 one another, and specifically urges that direct funding to private firms be compared 

 to the alternatives. One of the report's endnotes (endnote 37, page 37) illustrates 

 what the committee had in mind: 



Evaluation of investment programs to date has focused mainly on the question, 

 Would this technology ever have developed or would it have been significantly de- 

 layed but for the federal funding? Most assessments have been based on queries 

 to recipients and agency staff about judgments of success, and on limited meas- 

 ures of impact such as patent counts or financial measures that cannott answer 

 the question. What is needed is rigorous assessment through comparison to appro- 

 priate control cases. Moreover, answering one question does not address several 

 others that are equally important such as: How effective is direct federal invest- 

 ment in specific firms or consortia compared to investment in R&D through other 

 mechanisms, such as grants and contracts to do similar work at universities or 

 federal laboratories? Would incentives to R&D performers to ease start-up of new 

 firms or to encourage private investment through indirect means achieve the 

 same ends at less cost or with less direct federal involvement? How can direct in- 



1 See, for example, the following publications that were cited in the committee's report: Com- 

 puter Science and Telecommunications Board, National Research Council (1994), Realizing the 

 Information Future: The Internet and Beyond; Committee on Science, Engineering and Public 

 Policy (1993), Science, Technology, and the Federal Government: National Goals for a New Era, 

 chapter 4; National Science and Technology Council (1995), National Security Science and Tech- 

 nology Strategy; National Economic Council, National Security Council, and Office of Science 

 and Technology Policy (1995), Second to None: Preserving America's Military Advantage Through 

 Dual-Use Technology; Engineering Centers Division, Directorate for Engineering, National 

 Science Foundation, The ERCs: A Partnership for Competitiveness (NSF 991-9, 1991), Highlights 

 of Engineering Research Centers Technology Transfer (NSF 92-6, 1992), and Highlights of Engi- 

 neering Research Centers Education Program (NSF 95-56, 1995); NASA Federal Laboratory Re- 

 view Task Force, NASA Advisory Council (1995), NASA Federal Laboratory Review; Task Force 

 on Alternative Futures for the DOE National Laboratories (1995), Alternative Futures for the 

 Department of Energy National Laboratories; Department of Defense (1995), Department of De- 

 fense Response to NSTC / PRD #1, Presidential Review Directive on an Interagency Review of Fed- 

 eral Laboratories; Susan Rosegrant and David R. Lampe (1992), Route 128: Lessons from Bos- 

 ton's High-tech Community; AnnaLee Saxenian (1994), Regional Advantage: Culture and Com- 

 petition in Silicon Valley and Route 128; Economics Department, Bank of Boston (1989), MIT: 

 Growing Businesses for the Future; Barry Bozeman and Michael Crow (1995), Federal Labora- 

 tories in the National Innovation System: Policy Implications of the National Comparative Re- 

 search and Development Project; Ann Markusen et al. (1995), Coming in from the Cold: The Fu- 

 ture of Los Alamos and Sandia National Laboratories; In addition, the committee was informed 

 by many other publications, including, but not limited to: National Academy of Engineering 

 (1992), Time Horizons and Technology Innovations; National Academy of Engineering (1992), 

 Profiting from Innovation; Edwin Mansfield (1995), "Academic Research Underlying Industrial 

 Innovations: Sources, Characteristics, and Financing;" Gordon Bell (1994), "Government Role in 

 Technical Computing: Lessons from the United States," Decision Resources Spectrum; Gordon 

 Bell, "A Policy for Government Support of Computer Systems R&D: A Look at 50 Federally 

 Funded Computer Systems Research Projects Over 30 Years," manuscript sent by the author 

 in April 1994 with additions on February 22, 1995; MCC Ventures, Commercialization of Los 

 Alamos National Laboratory Technologies via Small Business, report to Los Alamos National 

 Laboratory LA-UR-94-1784 (Vol. 1) and LA-UR 94-1785 (Vol. 2); Jay Stowsky and Richard H. 

 White (1995), Anchoring U.S. Competitiveness: Revisiting the Economic Rationale for Technology 

 Policy; Advanced Research Projects Agency (1995), The Technology Reinvestment Project: Dual 

 Use Innovation for a Stronger Defense; Richard H. White et al. (1995), The Economics of Com- 

 mercial-Military Integration and Dual-Use Technology Investments; Richard H. White et al. 

 (1995), Assessing the Economic and National Security Benefits from Publicly Funded Technology 

 Investments: An IDA Roundtable; Richard H. Van Atta et al., DARPA Technical Accomplish- 

 ments, Volume I (1990), Volume II (1991), and Volume III (1991); NIST Visiting Committee on 

 Advanced Technology (1995), 1994 Annual Report; Solomon Associates (1993), The Advanced 

 Technology Program, An Assessment of Short-Term Impacts: First Competition Participants; 

 Silber & Associates (1995), Survey of ATP Pilot Program Awardees: Interim Report; National In- 

 stitutes of Standards and Technology (1994), Setting Priorities and Measuring Results at NIST; 

 and Government Accounting Office (1995), SBIR Interim Report. 



