Collaborative Expedition Workshop #73, Potentials and Realities of Certification in Light of Open Technology Development (3XKT)
Workshop Questions (3XKU)
A. People: Appreciation of Potentials with Open Technology Development / Expanding the Possibilities Through Decentralized, Transparent Structure / Setting the Tone for Open Inquiry, Experimentation and Social Exchange (3XIG)
- 1. What are the conducive conditions for the creativity and governance needed among networked scientific and scholarly communities so results and implications flow in a timely manner into legal, public policy, and national innovation preparedness channels? (3XIH)
- What are the current and future contributions of light-weight aggregator tools for advancing discovery, shared understanding, and organizing for stronger innovation capacity that scales across individuals, communities of practice, and institutions? How can these tools help us be individually accountable for collaborative actions relative to shared purpose? Examples in use by this workshop community include: wiki namesake pages, Emerging Technology Life-cycle process and Strategy Markup Language (StratML) (3XOE)
- How can public policy stakeholders tap Web 2.0 "build to share" principles being advanced by forward-looking information stewardship organizations in order to broaden common understanding of multi-faceted aspects of national/ global challenges and accelerate discovery of exemplary practices and innovative technologies? (3XOF)
- 2. How can effective roles and responsibilities be established for a myriad of professionals from multiple settings who need to respond to "overlapping" uncertainties with effective collaboration in an ever-expanding, data and information-rich world? (3XIM)
- Could these trusted communities contribute to collective advancements in discernment through controlled access to assurance cases: a) Digital data and information communities advancing sound approaches for electronically stored information(including librarians, curators, web content managers, ontologists, researchers, artists, historians, data managers, and records managers b) Open Standards bodies and consortia; c) Universities and university consortia; d) International stewardship associations; and e) Virtual organizations (3XJS)
- What are the changing roles and responsibilities around shared ownership, safety, security, licensing, etc. that need to be explicit and transparent (i.e. assurance case) in innovation framework agreements among multiple partners? (3XJV)
- What aspects of national innovation preparedness will most benefit from coordination that includes open dialogue in public settings? (3XIR)
- 3. How do we create commonly understood problem representations and simulations to help multiple disciplines and geographic regions build capacity for joint action that addresses our need for agile, safe, and comprehensive societal innovations? (3XIT)
- How do we provide the right sets of information flowing into and out of "what if" mission-policy simulations, etc. so understanding flows broadly even when the learning is experiential? (3XJ4)
- How can shared knowledge cyberinfrastructures emerging from multi-disciplinary efforts (e.g. public health, emergency management, public utilities, national innovation preparedness, and critical infrastructures) be validated and valued as public goods and national assets? (3XJ7)
- What are the implications of "open standards" and other forms of "openness" to global innovation processes? To trustworthiness of components? (3XIP)
- Are there approaches to preserving the artifacts of large-scale learning exercises and real events with sufficient integrity to leverage continuous improvements? (3XJ5)
- Can we see in the future of creative collaborative efforts (e.g. public health, urban planning, disaster management) a future that transcends what has passed? (3XIV)
- 4. What frameworks, representations, metrics, and procedures are needed for evaluating the trustworthiness of systems? (3XIK)
- What are the surrounding legal and insurance considerations? (3XIX)
- Are current legal and policy frameworks effective? Sufficient? (3XIY)
- What are the rights of citizens and institutions with respect to assurance case evidence and certification? (3XIW)
- How are builder-buyer-user risk and responsibility evaluated, allocated, declared? (3XIU)
- Is certification enabled by factored systems software mechanisms such as safety or security kernals, software interlock, encryption and key management services, and system support for capabilities? How would such mechanisms affect open systems certification? (3XJY)
- Do research investments in modular design representation, implementation, and reasoning have consequences for future design and certification practice? (3XJW)
- Do research results such as proof-carrying code, precise abstraction, behavioural interface theories, action logics, modular model checking and verification, and design and software synthesis have a place in compositional design, evidence management, and certification practice? (3XJX)
- What barriers must be removed to make research methods practical and widely used? What adaptation and integration is needed for their use in conjunction with less formal sources of evidence? (3XJZ)
B. Process: Transcending Insularity - Toward Transparency in Acquisition with Open Technology Deployment and Trusted Community Sharing and Exchange Networks (3XJ2)
- 1. How can the right information and information structures support effective technology evaluation, including assurance cases? (3XJA)
- How can information from multiple sectors and domains be right "at hand", "in place", and appropriately accessed, when needed to structure a cohesive assurance case that provides the right depth and breadth of information? (3XJB)
- What can be learned from new net-centric organizing principles in settings across multiple continuums (degree of formality, degree of familiarity, size, etc.? (3XOA)
- How can Open Technology Development principles and practices support better discernment, comparison, and evolution of distributed net-centric software modules in joint capability demonstrations and assurance cases? (3XNP)
- How can we accelerate multi-sector partnerships around net-centric capabilities needed for government to work in the likely scenarios of the 21st century? (3XNT)
- How could lightweight semantics developed through multi-stakeholder governance complement open format, robust storage, and fluid search to help integrate disparate information sources that support better national innovation preparedness? (3XJT)
- How could authoritative versions of existing policies, regulations, and legal procedures currently in place, be complemented by a "collective wisdom" version in order to broaden opportunities for suggested improvements, harmonization across boundaries, and creation of "synthetic" documents for easier comparison and constrast across institutions? (3XJU)
- How could lightweight semantics developed through multi-stakeholder governance complement open format, robust storage, and fluid search to help integrate disparate information sources that support better national innovation preparedness? (3XJT)
- 2. How might we expand the systems view of certification? (3XJC)
- Are changes needed in long-term research and transition investment strategies to create technology that would make design and certification more effective and less costly? (3XJH)
- Do we need new approaches for modeling and reasoning jointly about the computational and physical interactions of engineered systems? (3XJD)
- What are the implications of Toulmin formal argument structure as certification moves to assurance cases, beyond prescriptive check-lists? (3XKP)
- How are accident and incident reports tied back to certification evidence to detect gaps, flaws? Can and should they be? (3XK8)
- What disciplines should be involved? (3XJE)
- What educational deficits must be addressed relative to real-time systems, operation systems and systems software, continuous mathematics, dynamics and control, and concepts for safety engineering physical systems that are under cyber-control? (3XJG)
- Will current practice scale to emerging technologies, products? To component-based and model-based design? (3XK6)
- What about legacy systems? Ignore? Re-certify? Incrementally re-certify? Exploit demand for new function or technological capability to inject better certification methods and tools? (3XK7)
- Are changes needed in long-term research and transition investment strategies to create technology that would make design and certification more effective and less costly? (3XJH)
- 3. What are the similarities and differences associated with certifications of people (professional training), objects (i.e. medical devices), processes and scenarios that involve certified people, objects, and processes? (3XKQ)
- Is there an evolving role for ontologies and comprehensive metadata notation and annotation cababilities? (3XKR)
- Is there a role for initiatives like the Open Reputation Frameworks being developed by OASIS to improve the quality, transparency, and persistence of the risk communication that is required between players, ideally through dependability and trust cases? (3XKS)
C. Higher Performance, Greater Returns: Revisiting Total Cost of Ownership (TCO) in Contracting for High Confidence Assurance/ Amplifying Capacity through Sharing and Exchange across Communities of Practice and Communities of Interest (3XKC)
- 1. What properties are required of a certification methodology, tools, and results? (e.g., repeatability, comparability, assurance,...) (3XJI)
- Are there new needs for certification? (e.g. new security and privacy assurances? Need for new evaluated products?) (3XKK)
- Systems are engineered from components; how does (or should) certification take this into account? (3XKL)
- What are the cost and effectiveness trade-offs of post hoc certification vs. design-stage evaluation? (3XJJ)
- Are there design practice and evidence management approaches that might enable modular certification and re-certification? (3XJK)
- What level of understanding and foresight in contracting and acquisition is needed to achieve the desired level of re-use and agility? What is the right level of scoping - not too large (vendor lock-in) and not too small(hard to manage)? (3XNU)
- 2. What are the drivers of high certification/re-certification cost? (3XKD)
- 3. What is the state of current certification practices in various sectors? (3XKH)
- Can a roadmap be compiled for civilian agencies, modeled after DoD that will advance the transparent acquisition marketplace needed for increased mission agility? How well might this approach contribute to federated autonomy and "virtualization" in a manner that both addresses and transcends the concept of the Federal Government as a single, logical "enterprise" as suggested by Enterprise Architecture principles? (3XNW)
- What are essential understandings of sources of accidents, incidents, and failure? What best practice and directions in design can limit and mitigate these? (3XJ6)
- How can user interaction errors be avoided, detected, repaired? Should appropriate uses be certified? (3XJ9)
- 4. Are some approaches more cost-effective than others? (3XKJ)
- How can design practice and certification be better linked? (3XKM)
- Is certification effort directed towards the right problems and artifacts? (3XKN)
- What certification change processes are in use (e.g., NSA Ratings Mainenance Program, FAA re-certification by reference to prior evaluated system types)? Are these effective? What changes are needed? (3XKA)
- Is process evidence sufficient? (3XK4)
- Is certification effort directed towards the right problems and artifacts? (3XKN)
- Is certification-by-testing successful? (3XK5)
- Should certifications be revoked with cause? How can / should a technical basis be established and presented to manage this effectively? What are the consequences of revocation in large, long-lived systems? (3XK9)
- How can design practice and certification be better linked? (3XKM)