THE WASHINGTON RESEARCH EVALUATION NETWORK    (3WMM)

SPRING 2008 INTERACTIVE WORKSHOP    (3WKN)

Program Evaluation Methodologies, Policies, and Management of    (3WMN)

R&D Organizations: Where are We Now and Where do We Want to be?    (3WKO)

WREN Workshop Venue    (3WMO)

The George Washington University Marvin Center 3rd Floor Continental Ballroom H and 21st Street, NW, Washington, DC    (3WUO)

Friday, June 6, 2008    (3WUP)

8:15 - 8:45 am Arrival and Registration    (3WMS)

8:45 - 9:45 am Joint Plenary Session with the American Evaluation Association: ''The Role of AEA in Federal Evaluation Policymaking''    (3WMT)

AEA is reaching out to federal evaluators for ideas to better reach federal policymakers.    (3WKS)

What message should AEA convey to federal policy makers about the evaluation of federal programs and evaluation's usefulness, appropriate evaluation approaches, funding needed for performing evaluation and developing evaluation tools, among other important issues?    (3WKT)

Cheryl J. Oros, WREN Steering Committee, and Deputy Director, Clinical Science R&D Service, Veterans Health Administration, Department of Veterans Affairs Oros June 6 WREN Presentation    (3WKU)

William M.K. Trochim, 2008 AEA President and Chair, Evaluation Policy Task Force, Professor, Cornell University's College of Human Ecology, and Director of Evaluation of the Weill Cornell Clinical and Translational Science Center Trochim June 6 WREN Presentation    (3XSX)

Open Discussion with Audience, including:    (3WMU)

George Grob, AEA Evaluation Policy Task Force, Consultant, and President, Center for Public Program Evaluation    (3WKX)

Katherine Dawes, AEA Evaluation Policy Task Force, and Director, Evaluation Support Division, Office of Policy, Economics and Innovation, Environmental Protection Agency    (3WKY)

Concurrent Sessions 1 and 2    (3WMV)

9:45 - 11:00 am 1. Break-out Session to Continue AEA discussion - George Grob, Evaluation Policy Task Force, AEA, and Consultant and President, Center for Public Program Evaluation    (3WL0)

9:45 - 10:30 am 2. WREN Opening Plenary Session    (3WMW)

What progress have we made in the evaluation of science and technology research and development programs? Where we are headed and what efforts are currently underway in prospective (ex ante) evaluation? What are the top evaluation concerns and topics of current interest?    (3WL2)

How has PART changed the evaluation of Federal R&D programs and what will happen next? Are there upcoming evaluation events and key opportunities of which federal evaluators should be made aware?    (3WL3)

What training in R&D evaluation would federal evaluators find helpful? What training in R&D evaluation should be provided for OMB examiners, Congressional committee staff, and Senior Executive Service policy and decision makers?    (3WL4)

Cheryl J. Oros, WREN Steering Committee and Workshop Discussion Facilitator, and Deputy Director, Clinical Science R&D Service, Veterans Health Administration, Department of Veterans Affairs    (3WL5)

Gretchen Jordan, Co-Chair of AEA's Research, Technology, and Development Evaluation Topical Interest Group and Principal Member of Technical Staff, Sandia National Laboratories, Evaluation of R&D Programs: State of the Art Jordan June 6 WREN Presentation    (3WL6)

Stephanie Shipman, Federal Evaluators Coordinator and Assistant Director, Applied Research and Methods, Government Accountability Office, Approaches Used by R&D Agencies to Respond to PART and GPRA, and the Role of Evaluation in Oversight    (3WMX)

10:30 - 10:45 am Break    (3WMY)

10:45 - 12:00 pm Panel Discussion 1: ''How Can Prospective Evaluation Approaches Help Policymakers Determine Where in an R&D Portfolio to Invest?''    (3WL9)

Evaluation is more than a reporting exercise to examine the past - evaluation plays a vital role in informing future investments, especially in research organizations.    (3WLA)

How can we effectively use evaluation prospectively to help determine which scientific fields should receive greater future attention and funding, and to help balance a portfolio of research investments? How has the American Association for the Advancement of Science (AAAS) been involved? What is the status of the White House Office of Science and Technology Policy's (OSTP) "Science of Science Policy" Initiative? How does that effort relate to NSF's "Science of Science and Innovation Policy (SciSIP)"? How will research findings resulting from NSF SciSIP grants help policymakers formulate decisions on where discretionary funding should be spent?    (3WLB)

Connie Chang, WREN Steering Committee and Workshop Discussion Facilitator, and Director, Ocean Tomo Federal Services    (3WLC)

Bill Valdez, WREN Steering Committee, and Director, Office of Workforce Development for Teachers and Scientists, Department of Energy, Update on Science of Science Policy Initiative and Use of Prospective Evaluation Valdez June 6 WREN Presentation    (3WLD)

Julia Lane, Program Director, Science of Science and Innovation Policy, National Science Foundation, SciSIP Progress and Plans Lane June 6 WREN Presentation    (3WMZ)

12:00 - 1:15 pm LUNCH Provided to Promote Networking Among Federal evaluators, managers, methodologists, training providers, and others    (3WLE)

International WREN Partners are asked to provide a brief update of their key R&D evaluation efforts. Jang June 6 WREN Presentation    (3WLF)

Federal evaluators can meet and plan collaborations in evaluating joint R&D efforts, or plan joint efforts in fostering the development of new R&D evaluation approaches.    (3WLG)

Evaluators can meet methodologists to discuss possible evaluations and also training providers to discuss this year's offerings or customized needs.    (3WLH)

1:15 - 2:15 pm Panel Discussion 2: ''What is the Current State of Prospective (Ex-Ante) Evaluation Methodologies and How Can We Use Them?''    (3WLI)

A prospective (ex-ante) perspective has permeated recent approaches to evaluation in the United States in which questions are raised and directed at developing evaluation approaches, frameworks, methodologies, and thinking that help guide future decisions of where to invest our scarce R&D resources.    (3WLJ)

What new approaches and tools have been uncovered which are useful and applicable? Which agencies have used these tools? How can these tools be matched with different levels of decision making? Can systems-level analysis address gaps and assist in science and technology decision making at the federal level?    (3WLK)

Bhavya Lal, WREN Workshop Discussion Facilitator, and Senior Analyst for Metrics, Performance Measurement, and Strategic Planning, Science and Technology Policy Institute Lal June 6 WREN Presentation    (3XTP)

William M.K. Trochim, Professor, Cornell University's College of Human Ecology, and Director of Evaluation of the Weill Cornell Clinical and Translational Science Center, Evaluating R&D Programs Trochim June 6 WREN Afternoon Presentation    (3XTQ)

Paul Heisey, Economist, Resource, Environmental, and Science Policy Branch, Resource and Rural Economics Division, Economics Research Service, U.S. Department of Agriculture, Potential Roles for Ex-ante Economic Evaluation of Agricultural Research Heisey June 6 WREN Presentation    (3XST)

Nicholas Vonortas, WREN Steering Committee, and Director (on sabbatical), Center for International Science and Technology Policy and Professor, Department of Economics, George Washington University, 'Real Options' Framework to Assess Prospective Public Investments in R&D (project funded by DOE) Vonortas June 6 WREN Presentation    (3XSU)

2:15 - 2:30 pm Break    (3WLP)

2:30 - 3:45 pm Panel Discussion 3: ''What is the Current State of Evaluation Policy and Methodologies for Monitoring Program Performance?''    (3WLQ)

The President's Office of Management Budget and Congress have instituted two main program oversight efforts (PART and GPRA), but how have they actually influenced federal evaluation and program effectiveness and efficiency, and where will evaluation head next?    (3WLR)

How is the PART (OMB's Program Assessment Rating Tool) performing as a diagnostic tool to yield information on R&D program results? What methodologies are agencies using to monitor their performance? Have any best practices emerged? How has ExpectMore.gov contributed? What should agencies expect next?    (3WLS)

How well has GPRA (the Government Performance and Results Act of 1993) met its goals of holding agencies accountable for using resources wisely and achieving program results? What should agencies expect next?    (3WLT)

Stephanie Shipman, WREN Workshop Discussion Facilitator, and Assistant Director, Applied Research and Methods, Government Accountability Office    (3WLU)

Irwin Feller, Professor Emeritus of Economics, The Pennsylvania State University and Senior Visiting Scientist, American Association for the Advancement of Science, A Strategy for Assessing Science Feller June 6 WREN Presentation    (3XSV)

Phillip Juengst, Accountability Team Leader, Office of Research and Development, Environmental Protection Agency, Evaluating Research Efficiency in EPA (comments on National Academy of Science report) Juengst June 6 WREN Presentation    (3XSW)

Deborah Duran, Director, Systemic Assessments Branch, Office of Portfolio Analysis and Strategic Initiatives (OPASI), National Institutes of Health, NIH's Approach to GPRA and PART Duran June 6 WREN Presentation    (3WLX)

Timothy Hays, Director, Portfolio Analysis and Scientific Opportunities Branch, Office of Portfolio Analysis and Strategic Initiatives (OPASI), National Institutes of Health, NIH's Approach to Portfolio Analysis Hays June 6 WREN Presentation    (3WLY)

3:45 - 4:15 pm Adjourn and Networking Opportunity    (3WLZ)

Federal evaluators can meet and plan collaborations in evaluating joint R&D efforts, or to plan joint efforts in fostering the development of new R&D evaluation approaches. Evaluators can also meet methodologists to discuss possible evaluations and training providers to discuss this year's offerings or customized needs.    (3WM0)

* * * * * *    (3WN0)

The organizing team wishes to thank the Department of Veterans Affairs and The George Washington University for their generous financial support and Susan B. Turnbull, Senior Program Advisor, Intergovernmental Solutions, Office of Citizen Services and Communications, U.S. General Services Administration, GSA, for introducing WREN into the world of wikis.    (3WM2)

The co-sponsors wish to acknowledge and thank the organizing team,    (3WM3)

Cheryl Oros, Nicholas Vonortas, and Connie Chang,    (3WN1)

for their time, energy, and collaboration in launching this workshop.    (3WM4)

Important Upcoming Events:    (3WM5)

* * * * *    (3WN2)

Spring 2008 WREN Interactive Workshop    (3WMC)

Co-sponsored by:    (3WMD)

The Washington Research Evaluation Network (WREN) serves as a forum for the Federal R&D evaluation community to explore new approaches that will improve the management of science and technology organizations. Key questions addressed by WREN include: What is the overall justification for continued federal investments in science and technology, as well as the basis for specific investments in various science areas? How can the outcomes of R&D investments be systematically assessed? WREN brings together evaluation practitioners from the Federal government with theorists from academia and experts from private industry to develop evaluation approaches and fundamentally new theories about the behavior of R&D/S&T organizations. WREN is part of a larger international evaluation network of members in other countries that have similar interests in R&D evaluation, including our colleagues in Canada, Europe, Japan, Korea, New Zealand, among others. For more information, please visit: www.wren-network.net.    (3WMF)

The American Evaluation Association (AEA) is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness. AEA has approximately 5,000 members representing all 50 states in the United States as well as over 60 foreign countries. Find further information at www.eval.org. AEA's mission is to:    (3WN9)

AEA's Research, Technology, Development (RTD) Evaluation Topic Interest Group (TIG) is one of more than 40 TIGS defined around a special topic. The RTD TIG is very active and international in scope. For more info, please contact Gretchen Jordan, gbjorda@sandia.gov, Brian Zuckerman, bzuckerma@ida.org, or George Teather, gteather@sympatico.ca.    (3WMH)

Washington Evaluators (WE) is the local Washington area affiliate of the American Evaluation Association. Further information can be obtained at www.washeval.org.    (3WMI)

Federal Evaluators (FedEval) is an informal association of evaluation officials in the legislative and executive branches of the U.S. government. The group was organized to share information and concerns about evaluation methodology, policy, and practice at the federal level. There are over 160 members from all 15 cabinet departments and several independent agencies. The group meets several times a year to discuss issues of topical interest and has a "closed" listserv, FEDEVAL, for discussion of program and policy evaluation and related issues. Further information can be obtained at www.fedeval.net.    (3WMJ)

The Department of Veterans Affairs' Veterans Health Administration's Clinical Science Research and Development Service (CSR&D) supports and conducts research that focuses on intact human beings as the unit of examination, including interventional and effectiveness studies, clinical, epidemiological and technological studies. CSR&D was created through the reorganization of the former Medical Research Service and Cooperative Studies Program. For more information, please visit: http://www.research.va.gov/programs/csrd/.    (3WMK)

The George Washington University, Center for International Science and Technology Policy conducts cutting-edge research on the policy issues that affect science and technology around the world. Through major international conferences and seminars, the Center facilitates and inspires collaboration among policy experts and government and industry leaders on science and technology issues. The Center collaborates with international governmental research institutes and agencies on advances in scientific and technological policymaking. The Center also hosts visiting scholars from around the world and offers a Master of Arts degree in International Science and Technology Policy. The Center also houses GWU's Space Policy Institute, which concentrates its research and instruction on issues related to the United States and global space programs, including human space flight, space transportation, earth observations from space, and space law. For more information, please visit: http://www.gwu.edu/~elliott/academicprograms/ma/istp/index.cfm.    (3WML)