SlideShare a Scribd company logo

US University Research Funding, Peer Reviews, and Metrics

•Download as PPTX, PDF•
•
Daniel S. Katz
Daniel S. Katz

My part of the "Digital Science Webinar: Articulating Research Impact – Strategies from Around the Globe" (http://www.digital-science.com/events/digital-science-webinar-articulating-research-impact-strategies-from-around-the-globe/) Daniel S. Katz will discuss how reviewers at the National Science Foundation (USA) consider the “intellectual merit” and “broader impacts” criteria for funding and in particular how metrics might help applicants understand their impacts in these areas.Dan will also talk about how reviewers might use qualitative and quantitative altmetrics data to inform their peer reviews for grant applications. He will address many of the salient questions around this use of metrics, for example, do reviewers take metrics seriously and what types of metrics are of most value to them?

Related slideshows

Theory-Based Approaches for Assessing the Impact of Integrated Systems Research
Theory-Based Approaches for Assessing the Impact of Integrated Systems ResearchTheory-Based Approaches for Assessing the Impact of Integrated Systems Research
Theory-Based Approaches for Assessing the Impact of Integrated Systems Research
 
Ron Haskins Evidence based policymaking uk
Ron Haskins Evidence based policymaking ukRon Haskins Evidence based policymaking uk
Ron Haskins Evidence based policymaking uk
 
Esa Group Work System Approach in Flagship
Esa Group Work System Approach in FlagshipEsa Group Work System Approach in Flagship
Esa Group Work System Approach in Flagship
 
1 of 8
US University Research Funding, Peer Reviews, and Metrics Daniel S. Katz Computation Institute, University of Chicago & Argonne National Laboratory @danielskatz d.katz@ieee.org
US University Funding • Large number of universities – Public (state-funded, not federally-funded), private, for profit • No direct national funding • Indirect national funding of education through students • Indirect national funding of research through projects Research assessment is specific to projects, usually as part of proposals
National Research Funding Agencies • National Science Foundation – Mission: support for all fields of fundamental science and engineering, except for medical sciences • National Institutes of Health – Mission: to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability • Department of Energy – Mission: to ... [address] energy, environmental and nuclear challenges through transformative science and technology solutions • Department of Defense – Mission: ... to protect the security of our country • Many other agencies – Each with a specific mission Research assessment is specific to agencies, usually as part of proposals
NSF Standard Merit Review Criteria Intellectual Merit Broader Impacts What is the potential for the proposed activity to advance knowledge and understanding within its own field or across different fields? What is the potential for the proposed activity to benefit society or advance desired societal outcomes? To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? How well qualified is the individual, team, or organization to conduct the proposed activities? How well qualified is the individual, team, or organization to conduct the proposed activities? Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities? Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities? http://www.nsf.gov/bfa/dias/policy/merit_review/ Intellectual Merit What is the potential for the proposed activity to advance knowledge and understanding within its own field or across different fields? To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? How well qualified is the individual, team, or organization to conduct the proposed activities? Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities?
Additional Review Criteria • NSF Software Infrastructure for Sustained Innovation (SI2), for example, includes: – Does the proposal discuss how the proposed software will fill a recognized need and advance research capability within a significant area (or areas) of science and engineering? – Are tangible metrics described to measure the success of any software that may be developed? – Does the software engineering and development plan include and/or enable the integration of relevant research activities to ensure the software is responsive to new computing developments? http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf14520
Agencies Expectations for Societal Benefit • NSF – 2 non-scored criteria (one is broader impacts), plus maybe more, put together qualitatively, one final rating • NIH – Scored criteria put together quantitatively, plus non-scored criteria – No specific request for description of societal benefit, but need is strongly implied in all programs • DOE and other mission agencies, impact is measured in how research contributes to their missions – Non-scored review processes in general – At a high level, succeeding in this missions already includes the concept of societal benefit
Determining What NSF Funds • Peer review of new project is primary factor – Also includes proposers information about previous projects • What is the role of data? • As input to peer review, not to replace it • So, the question is what data will give the peer-reviewers new information and potentially lead them to different decisions – Answers may be project specific
Guidance to Proposers - Metrics • Decide what you want to measure to make a convincing case for your project – Quality – Impact – Attention • Then decide how to measure it – Technical metrics – Biobliometrics – Altmetrics

More Related Content

US University Research Funding, Peer Reviews, and Metrics

  • 1. US University Research Funding, Peer Reviews, and Metrics Daniel S. Katz Computation Institute, University of Chicago & Argonne National Laboratory @danielskatz d.katz@ieee.org
  • 2. US University Funding • Large number of universities – Public (state-funded, not federally-funded), private, for profit • No direct national funding • Indirect national funding of education through students • Indirect national funding of research through projects Research assessment is specific to projects, usually as part of proposals
  • 3. National Research Funding Agencies • National Science Foundation – Mission: support for all fields of fundamental science and engineering, except for medical sciences • National Institutes of Health – Mission: to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability • Department of Energy – Mission: to ... [address] energy, environmental and nuclear challenges through transformative science and technology solutions • Department of Defense – Mission: ... to protect the security of our country • Many other agencies – Each with a specific mission Research assessment is specific to agencies, usually as part of proposals
  • 4. NSF Standard Merit Review Criteria Intellectual Merit Broader Impacts What is the potential for the proposed activity to advance knowledge and understanding within its own field or across different fields? What is the potential for the proposed activity to benefit society or advance desired societal outcomes? To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? How well qualified is the individual, team, or organization to conduct the proposed activities? How well qualified is the individual, team, or organization to conduct the proposed activities? Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities? Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities? http://www.nsf.gov/bfa/dias/policy/merit_review/ Intellectual Merit What is the potential for the proposed activity to advance knowledge and understanding within its own field or across different fields? To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts? Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success? How well qualified is the individual, team, or organization to conduct the proposed activities? Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities?
  • 5. Additional Review Criteria • NSF Software Infrastructure for Sustained Innovation (SI2), for example, includes: – Does the proposal discuss how the proposed software will fill a recognized need and advance research capability within a significant area (or areas) of science and engineering? – Are tangible metrics described to measure the success of any software that may be developed? – Does the software engineering and development plan include and/or enable the integration of relevant research activities to ensure the software is responsive to new computing developments? http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf14520
  • 6. Agencies Expectations for Societal Benefit • NSF – 2 non-scored criteria (one is broader impacts), plus maybe more, put together qualitatively, one final rating • NIH – Scored criteria put together quantitatively, plus non-scored criteria – No specific request for description of societal benefit, but need is strongly implied in all programs • DOE and other mission agencies, impact is measured in how research contributes to their missions – Non-scored review processes in general – At a high level, succeeding in this missions already includes the concept of societal benefit
  • 7. Determining What NSF Funds • Peer review of new project is primary factor – Also includes proposers information about previous projects • What is the role of data? • As input to peer review, not to replace it • So, the question is what data will give the peer-reviewers new information and potentially lead them to different decisions – Answers may be project specific
  • 8. Guidance to Proposers - Metrics • Decide what you want to measure to make a convincing case for your project – Quality – Impact – Attention • Then decide how to measure it – Technical metrics – Biobliometrics – Altmetrics