• A single review by a panel, which passes the 

 summary evaluation of the proposal to the 

 program manager, who considers the advice 

 and recommends action; 



• A dual review system in which more than 

 one panel evaluates the proposal sequentially 

 (as in the NIH system, where the Study Sec- 

 tion first ranks the proposals and an Advi- 

 sory Council reviews these ratings and ap- 

 proves or declines the proposal on the basis 

 of its relevance to the agency's plan); 



• Mail review, whereby the proposal is mailed 

 to reviewers who are experts in particular 

 fields and a program manager then recom- 

 mends action based on the review; and 



• Program manager review, whereby the pro- 

 gram manager has total responsibility for 

 seeking advice (often obtained through per- 

 sonal contact with experts) and making the 

 final decision.'' 



In special cases, when dealing with difficult or 

 controversial decisions, some agencies may use 

 several of these methods. There is a wide diver- 

 gence of opinion among agencies and program 

 managers on the relative value of the above pro- 

 cedures; the decision to adopt one or another is 

 often based on customs in a field of learning as 

 well as on agency programs and management pat- 

 terns. 



Decisions to terminate basic research programs 

 are based on the same data and analysis as deci- 

 sions on project renewals or extensions. Because 

 progress in applied research usually depends on 

 advances in basic research, agencies often evalu- 

 ate the two at the same time. Almost every agen- 

 cy has adopted evaluation techniques, using one 

 or a combination of visiting committees, panels, 

 and consultants to assist management in deciding 

 on project continuation. The National Bureau of 

 Standards (NBS) uses four principal means for 

 evaluating its research programs: (I) internal 

 management reviews at each level in the Bureau; 

 (2) a contract with the National Academy of Sci- 

 ences to provide independent review and evalua- 

 tion (29 panels are involved); (3) Bureau survey 

 and evaluation by its statutory visiting committee; 

 and (4) comprehensive analyses of issues in se- 

 lected programs by qualified contractor personnel. 

 (A more detailed description of this procedure is 

 included in the NBS material in Part I.) 



ONR relies on continuing site visits by the pro- 

 gram managers and representatives from regional 



'For a detailed discussion of review procedures, ttie reader 

 is referred to Wirt. John C, Liebierman. Arnold J., and l.ev- 

 ien, Roger E.. RdD Manugement: Methods Used by Fcder:il 

 Agencies (The Rand Corporation; Santa Monica, Calif), 1974. 



offices; NASA uses an annual review system that 

 includes extensive detailed reports; NSF uses site 

 visits and reviews of proposals for continuing 

 support, together with such panel evaluations as 

 may be available. For its national facilities, NSF 

 requires that the contractor's governing board 

 examine all programs and activities on a regular 

 schedule, using such advisory or review commit- 

 tees as may be appropriate. These appraisals are 

 reported to NSF on a schedule corresponding to 

 the presentations of plans, programs, and budgets 

 for subsequent years. In addition, the Director of 

 NSF has established a program evaluation staff, 

 which, through outside contract or by other as- 

 sistance such as special consultants or ad hoc 

 panels, seeks to evaluate entire programs. The 

 National Science Board also performs a major 

 role in the evaluation and approval of ongoing 

 programs. 



Regardless of what operational mechanisms an 

 agency chooses for evaluating ongoing projects, 

 the result is a determination that the work either 

 should continue or be terminated. If the review 

 suggests that the work probably will not yield re- 

 sults within a time frame the agency can accept 

 (as is often the case when basic research is initiat- 

 ed to support some applied project), a panel often 

 will recommend termination. Such recommenda- 

 tions generally pass through several layers of 

 management. An examination of agency practices 

 indicates that such terminations are more likely to 

 occur when program funds are limited. Sometimes 

 a project is selected for termination because such 

 action is the only way to initiate fresher pro- 

 grams. Other reasons for termination include lack 

 of qualified personnel and pressure to eliminate 

 programs with long-term payoffs. By various 

 means, then, agencies continuously examine and 

 evaluate their new and ongoing basic research 

 projects, continuing or terminating them in ac- 

 cordance with their mission objectives. 



Criteria for Determining Levels of Funding 

 and Sites Where Research Is Done 



The agency-approved research plan influences the 

 level of funding given a project and the selection of 

 the laboratory or site at which it will be carried out. 

 In selecting the material to be included in the plan, 

 some bias is established which cannot be overlooked 

 when the plan is carried out. 



Decisions on levels of funding are influenced by 

 several criteria, including; (I) the availability of 

 funds in optimum or at least adequate amounts to 

 support a reasonable level of research; (2) the 

 probability that the general program or project 

 will be successful; (3) the currentness of the re- 

 search as determined by the literature and by pro- 



AGENCY SUPPORT OF BASIC RESEARCH BY FIELD OF SCIENCE 269 



