SOA Predictive Metrics 
John Salasin, Ph.D.
Visiting Researcher
NIST
jsalasin@nist.gov

This project is specifying “Predictive Metrics” -- measures (characteristics of the application and technology) that can be made early in the life cycle based on models of the system (technology and applications(s)) and will influence success in later stages. The models/metrics should suggest likely problems / opportunities at future stages of development / deployment to suggest corrective action.

The Predictive Metrics should provide a basis for continual system-level, Verification & Validation (V&V), in contrast to fixed milestone-based assessments of whether (paper) products have been delivered. The metrics should be useful in deciding if there is, at any point in its development/evolution, a high probability that the system will meet its requirements – and a minimal risk that it won’t.

Since an estimated 2/3 of a systems’ cost comes after its initial deployment, many of the models/metrics represent factors that are likely to influence Total Cost of Ownership (TCO). A central principle of an SOA-based system is that the services and their orchestration map directly to business processes which often change every 12-18 months,. The ease of synchronizing a business process change with the SOA, which is strongly related to the notation(s) used to express each, can have a major and continuing cost impact. If the translation has to be done manually, costs are likely to increase dramatically and translation errors to be introduced.

While our models/metrics focus on technical, rather than management or governance factors (as the SEI’s CMM), different organizations, programs and systems differ in the sophistication of the approaches they use for sharing information, engineering services, or increasing their value. We are adopting work done at the USDOJ to reflect “technical maturity” in the models.

The project considers metrics related to Comprehensiveness, Utility, Extensibility, and Technology Infrastructure at three broadly defined and often overlapping stages;
•	Early stage – concept development, obtaining management support. The functions of the system and interaction with other organizations are specified in general. We can estimate the scope of the services provided in terms of the number and types of consumers – from a single program to the public at-large.
•	Architecture/Construction stage -- a number of SOA components are being specified and prototyped/implemented. Models produce metrics predicting the impact on business results from the IT initiative.
•	Operations stage -- measurements / models initiated in previous stages are validated/refined. The focus is on capturing (rather than estimating) key elements of performance that directly relate to the IT initiative.
•	Evolution – collects data to validate models of the ease with which a system can be modified. In addition, evolution includes the expansion of a system to include additional organizational units in the enterprise.

This is a work in progress.  We are eliciting comments and suggestions from Government and Industry on:
•	The validity of the indicators – are they predictive? Are some irrelevant?
•	Better ways of quantifying the indicators;
•	Anecdotal examples of situations where the existence of a condition has (or has not) had a favorable (or unfavorable) impact;
•	How a metrology developed around these indicators/metrics might be most useful in Program/Project management – particularly with respect to ongoing Verification and Validation (V&V).
•	Experiments that might provide benchmark data indicating expected ranges for different values and predict their impact on system performance, utility, and TOC.