Past and Future Collaborative Expedition Workshops    (3VUK)

Collaborative Expedition Workshop #71, March 18, 2008, at NSF    (3VUL)

4201 Wilson Blvd., Room 1235 NSF Board Room    (3VUM)

Technology Program Evaluation: Methodologies from the Advanced Technology Program    (3VUN)

A. Workshop Purpose    (3VUU)

This workshop provides an opportunity for participants to better understand how to evaluate the social and economic benefits from public and private investments in emerging technology. The compelling role of sound economic transaction analysis (including credible commitments, risk mitigation, and foresight in contracting) and best practices for technology evaluation will be shared, based on the former Advanced Technology Program, a globally-recognized program at the National Institute for Standards and Technology (NIST). What can Communities of Practice learn about technology evaluation, risk mitigation, and foresight in contracting, from this globally renowned program? (The ATP was concluded in August 2007 under the America Competes Act).    (3VUV)

The workshop sessions will be led by NIST personnel from the former ATP Economic Assessment Office (EAO) focusing on the methodologies developed to evaluate ATP funded projects in light of the program's mandate to encourage industry to accelerate the creation and commercialization of enabling technologies that are expected to yield large economic benefits to the nation, extending significantly beyond the direct benefits to award recipients. The workshop will also include contractors who have played a significant role in developing and applying evaluation metrics.    (3VUW)

Evaluation consisted of a multi-faceted approach including: data collection, through project management and surveys; data analysis, employing statistical and economic investigation conducted both internally and externally; case studies, incorporating an evaluation metric applied to completed projects; and, dissemination of the program’s achievements, through academic and professional publications and presentations.    (3VUX)

In addition this workshop is timely for participants who have come to appreciate that building trusted relationships is the essence of eGovernment. How people design the organizing processes for potentially "collaborative" settings is a topic covered in several ATP papers through its surveys and studies.    (3VUY)

Advanced Technology Program (ATP) Background    (3VV5)

The Advanced Technology Program’s (ATP) at the National Institute of Standards and Technology was a partnership between government and private industry to conduct high-risk research to develop enabling technologies that promise significant commercial payoff and widespread benefits for the economy. Since the inception of the ATP in 1990, ATP’s Economic Assessment Office (EAO) performed rigorous and multifaceted evaluations to assess the impact of the program and to estimate the returns to taxpayers. To evaluate whether a project and portfolio of projects met it stated objectives, EAO used several methodological approaches to measure program effectiveness. These methods assessed progress over the lifetime of the project in terms of:    (3VV6)

Key features of ATP’s evaluation methods will be presented at this workshop. We hope that the methodologies presented today will inform other evaluation programs.    (3VVB)

B. Workshop Issues    (3VVC)

All organizations are accountable to someone, and the persons or organizations to which they are accountable are called stakeholders. Stakeholders generally offer their support to a program on the condition that their targeted goals and objectives are met, and stakeholders usually require evidence that they are getting the desired return on their investment. Stakeholders in private organizations are stockholders and other owners and investors, with employees, managers, and boards of directors also in stakeholder roles. While stakeholders in public agencies are ultimately the citizens, the legislative and executive branches are the de facto stakeholders. Other individuals and groups, including agency employees, program administrators, oversight committees, program participants, and the communities from which these participants are drawn, also have stakeholder roles. Evaluation can be used as a tool for stakeholders to learn if their objectives are being met.    (3VVD)

Evaluation can also lead to:    (3VVE)

Why evaluate?    (3VVI)

C. Expected Participants    (3VVO)

D. Agenda    (3VVP)

8:30am - Check-in and Coffee    (3VVQ)

8:45am - Welcome and Overview . [ slides ] . [ audio ]    (3VVR)

SusanTurnbull, GSA, Co-chair, Emerging Technology Subcommittee, and Co-chair, Social, Economic and Workforce Implications of IT and IT Workforce Development Working Group, Subcommittee on Networking and Information Technology Research and Development (“NITRD SEW”)    (3VVS)

RichardSpivack, NIST, Co-chair, Emerging Technology Subcommittee, and Economist, Impact Analysis Office Technology Innovation Program, NIST    (3VVT)

9:00am - Introductions: Attendee self introductions & brief statements of interests in light of the workshop purpose . [ audio ]    (3VVU)

9:20am - Technology Program Evaluation: Methodologies from the Advanced Technology Program, RobertSienkiewicz, (NIST) and RichardSpivack, (NIST) . [ slides ] . [ audio ]    (3VVV)

9:40am - Starting an Evaluation Program: Challenges and Lessons Learned, RosalieRuegg, (TIA Consulting) . [ slides ] . [ audio ]    (3VVW)

10:00am - Break    (3VVX)

10:15am - Panel Session 1: Survey and Data Infrastructure    (3VVY)

Survey Data collection - JenniferOBrien (Westat) . [ slides ] . [ audio ]    (3VW0)

Survey Development and Uses - StephenCampbell (NIST) . [ slides ] . [ audio ]    (3VVZ)

Managing Database Integration - TedAllen (5x5 Marketing Systems) . [ slides ] . [ audio ]    (3VW1)

Discussion . [ audio ]    (3VW2)

11:15am - Panel Session 2: Research Use of Data    (3VW3)

Research Applications of Data - AndrewWang (NIST) . [ slides ] . [ audio ]    (3VW4)

Data Enclave - TimMulcahy (NORC) . [ slides ] . [ audio ]    (3VW5)

Discussion . [ audio ]    (3VW6)

12:15 - 1:15pm - Lunch    (3VW7)

1:15pm - Panel Session 3: Status Reports and Gems    (3VW8)

Status Reports: Measuring against Mission - LeeBowes (NIST) and StefanieCox (Pricewaterhouse Coopers) . [ slides ] . [ audio ]    (3VW9)

ATP Gems: Communicating Impacts to an External Audience - PrasadGupte (NIST) . [ slides ] . [ audio ]    (3VWA)

Discussion . [ audio ]    (3VWB)

2:00pm - Panel Session 4: Case Studies and Analyses    (3VWC)

Cost-Benefit Analysis Studies - JeannePowell (Economic Consultant) . [ slides ] . [ audio ]    (3VWD)

New Ways to Assess Innovation in ATP - ConnieChang (Ocean Tomo Federal Services, LLC) . [ slides ] . [ audio ]    (3VWE)

Oct. 2007, The Emerging Clusters Project, Dept. of Commerce    (3WF7)

Novel Methodologies: 1) the relation between spillovers and research networks, 2) measuring the innovative capacities of different regions - KathyMcTigue (NIST) . [ slides ] . [ audio ]    (3VWF)

Discussion . [ audio ]    (3VWG)

3:00pm - Summary and Concluding Remarks - StephanieShipp (NIST, STPI) . [ audio ]    (3VWO)

3:30pm - Upcoming Workshops - SusanTurnbull, RichardSpivack . [ audio ]    (3VWP)

3:35pm - ADJOURN    (3W3V)

E. Additional Resources    (3VWQ)

ATP papers that adddress collaborative organizing    (3VWU)

Emerging Technology Life-cycle Program of the Emerging Technology Subcommittee - http://et.gov, JimDisbrow, Program Manager    (3VWY)

Technology Innovation Program, NIST - The Technology Innovation Program (TIP) was established for the purpose of assisting U.S. businesses and institutions of higher education or other organizations, such as national laboratories and nonprofit research institutions, to support, promote, and accelerate innovation in the United States through high-risk, high-reward research in areas of critical national need - http://www.nist.gov/tip/    (3VYL)

F. Workshop Series Background    (3VWZ)

Purpose and Audience: GSA's USA Services/ Intergovernmental leads monthly Collaborative Expedition workshops to advance the quality of citizen-government dialogue and collaborations at the crossroads of intergovernmental initiatives, Communities of Practice, Federal IT research and IT user agencies. The workshops seek to advance collaborative innovations in government and community services such as emergency preparedness, environmental monitoring, healthcare and law enforcement.    (3VX0)

The workshops serve individuals from government, business, and non-government organizations to practice an emerging societal form, Intergovernmental Communities of Practice (CoPs), in light of the Citizen-Centric Government goal of the President’s Management Agenda and the Public Information Access provisions of the E-government Act of 2002.    (3VX1)

Each workshop organizes participation around a common purpose, larger than any institution, including government. By learning how to appreciate multiple perspectives around potentials and realities of this larger “purpose”, subsequent actions by individuals representing many forms of expertise, can be better expressed in their home and collaborative settings. By centering around people and the "whole system" challenges they organize around, IT design and development processes can mature with less risk and greater national yield of breakthrough performance.    (3VX2)

Joint workshop sponsors in addition to GSA, include the Emerging Technology Subcommittee of the Architecture and Infrastructure Committee and Coordinating Groups of the Subcommittee on Networking and Information Technology Research and Development, including, Social, Economic and Workforce Implications of IT and IT Workforce Development CG, High End Computing CG, High Confidence Software and Systems CG, Software Design and Productivity CG, and Human-Computer Interaction and Information Management CG. These organizations value this “frontier outpost” to open up quality conversations, augmented by information technology, to leverage the collaborative capacity of united, but diverse sectors of society, seeking to discover, frame, and act on national potentials.    (3VX3)