62 



l\ll'K()\l\(, IIILAIJAXAnoS PROCESS/ ,i~ 



•%" Evaluation ot invc-Miiicnt programs to date has tbciiscd mainly on the question. Would this 

 ictlinoloji) e\er have ck-vcioped or would it have been sii;nificantly delayed but for the federal 

 lundinn:' Most assessments ha\e been based on queries to recipients and agency staff about judg- 

 ments of success, and on limited measures of impact such as patent counts or financial measures that 

 cannot answer the c|ueslion. VC hat is needed is rigorous assessment through comparison to appropri- 

 ate control cases. Moreover, answering one question does not address several others that are ecjually 

 important, such as: Mow effective is direct federal investment in specific firms or consortia com- 

 pared to investment in K&I) through other mechanisms, such as grants and contracts to do similar 

 work at universities or federal laboratories'' Would incentives to R&D performers to ease start-up of 

 new firms or to encourage pri\aie investment through indirect means achieve the same ends at less 

 cost or with less direct federal involvement? How can direct investments in firms or consortia confer 

 proprietarv advantage and \ei ensure public accountabilitv and fair access by other firms to data, 

 results, and expertise' 



38. NSr,\dvisor\ Committee on .Merit Keview. I'iiuil Report. NSF 86-9.3 (Washington. D.C: 

 National Science I'oundaiion. iy8()) 



.■iy Dorman Kepon IWS;(,al\in Report. IWS: foster Repon. 1995; Bishop/Calabresi Repon. 

 l99S:and Cassell/.Marks Kepon. 199-4. 



-to Re\ lew s of federal laboratories consistentiv conclude that procedures for judging the 

 c|ualit\ of research are not adequate and in practice do not have much effect on the allocation of 

 research funding See. for example. Defense Science Board. Labaratoiy Management Interim 

 Hc/xirl. 1995 (note 10); MishopA alahresi Report. 1995; Foster Repon. 1995; Casscll/Marks Report. 

 I9')-|. .\;itional Research Council. Inlerhii Repitrt of the Committee on Research and Peer Review 

 ill i'l'A (Wasliinglon. D ( National .Academy Press. 1995); Carnegie Commission on Science, 

 lechnologv. anil doxernnient. I'.iiriroiiiiieiildl Research and Development: Strengthening the 

 letleral In/raslnicliiiv (.New >ork: (!arnegie Commission on Science. Technology, and Government, 

 I99J); r ^ lln ironmenial Protection Agency. .Si//ei;;(</rr//;(,i,' the Future: Credible Science. Credible 

 Decisions. i:i'.\/(>(l()/M-')l/()S(). Expen Panel on the Role of Science at EPA (Washington, D.C.; U.S. 

 (lovernment Priming Office. l99-() 



-1 1 See. lor e\;uTiple. National Research Council. Investing in the \ational Research Initia- 

 tive: .\ii I pdtiiv III the ( onipciilive Crtinls Program in the I 'S. Department of .Agriculture 

 (Washington. I) ( N;itional Ac;idem\ Press. 199-0 



-i2 Misan I: ( i)//ens. ■.\ssessment of Fundamental Science Programs in the Context of the 

 (lOvernment Performance and Results Act ((iPR,\).' R.\.ND Project Memorandum PM-»l"'-OSTP 

 (Washington. D.C.: CrilicalTechnologies Institute. 1995) 



h3 Thomas 1) ( ook andWllliam R. Shadish. Program Evaluation; The Worldly Science." 

 AiniiKil Reiieiis of Psychology 3~; 19.3-2.32. 198(k Robert K. .Merton. The Sociology of Science: 

 Theoretical and limpirical Investigations (('hicago: llniversir\- of Chicago Press. 19''3); PH. Rossi. 

 ILL. Freeman, anil S. Rosenbaum. /rr^////^///o;;; .-f .Systematic .Approach (Beverly Hills. Calif.; Sage 

 Publications. 1982) 



-1-1 (.oz/ens. Assessment of Fundamental Science Programs.' 1995; Commission on Physical 

 Sciences. .Mathem;iiics. and .Applications of the National Research Council. Quantitative Assessments 

 III the Physical and Mathematical Sciences: .A Summary of Lessons Learned (Washington. D.C.; 

 National Academx Press. l')9-i); Susan E (^ozzens. rapponeur. £rrt//<tf^;o;; of Fundamental Research 

 Progrcniis: .A Revieiv ol the /.v.\/«'.v. discussion draft. Office of Science and Technology Policy. August 

 15. 199-1; and Sus;ui Coz/ens. Steven Popper lames Honomo. Kei Koizumi, and Ann Flanagan, 



