Step 4, the final step of the Solution Search Path, is to evaluate the options on the menu, make improvements as appropriate, and then select the desired solution.
Evaluation
Evaluation is about ranking the solution options on the menu against objective discriminators that are understood by and acceptable to the stakeholders. Keep the list of discriminators as short as possible. Use only the ones that strongly discriminate among the options. The evaluation ranking scores are by their nature not precise, so it adds no value to the evaluation process to include lessor discriminators. It only adds to the work.
Choosing discriminators
- Convert needed and desired qualities of outcome in the Definition of Success into quantifiable Measures of Effectiveness.
- Pick the ones that are likely to be strong discriminators. That is, are among the more important ones in determining overall outcome, and are likely to score differently on different solution options.
- Set up the analytics that convert features of any solution option into capability to deliver on the Measures of Effectiveness that were chosen.
What if two or more options come out close in the rankings? One way to resolve this is to realize that, if solutions evaluate similarly, it most likely doesn’t make a difference, so just flip a coin. Another approach is to look for more discriminators to break the tie.
Criteria
Criteria for evaluation of options are derived from the Definition of Success. One simple way to proceed is described below, or other methods may be devised.
- For each of the qualities of outcome in the Definition of Success, define a Measure of Effectiveness.
- Then establish a metric by which that Measure of Effectiveness is quantified. Ideally, that metric will be a parameter with values on a numerical scale. However, it is often the case that a Measure of Effectiveness is a qualitative property. That’s particularly so for Human Activity Systems. In such a case, a suitable approach is to identify examples which establish a ranking scale.
- For each Measure of Effectiveness, establish a pass-fail threshold. Those options that fail are immediately ruled out.
- Above the pass-fail threshold, assign points for how far the option exceeds the threshold, weighted by how important the Measure of Effectiveness being evaluated is to the overall success of the solution.
- Compute the aggregate score for each option.
Analytics
Analytics play a key role in this process. Determining the value of each metric for a particular option is an analytic process. See the branch on Analytics under Untangling Complexity.
This first pass at evaluating all the options does not complete this step. The evaluation needs a sanity check to determine whether the evaluation process has really addressed all the distinguishing factors that separate the various options, and whether the methodology really makes sense from a big-picture perspective.
Improvement
Further, the possibility of improvements to the options should be examined. Individual options may be boosted up in the ranking by making modifictions to them, or new, better options may be created by mixing contributions from various options on the menu.
Finally, the whole process needs stakeholder review, with corrections and improvements made in response to stakeholder critique.
Selection
When all concerns have been satisfied with stakeholder consensus support, the front-runner is selected as the preferred solution.
Project Review
Step 4 is also the place to step back and review the entire project. Has all the necessary work been done, and done well? If there are any soft spots, go back and bring them up to snuff. Does everything pass the smell test, that is, from the overall point of view, does it make sense and are all the conclusions plausible and well-founded? Are the stakeholders, by and large, happy with the outcome, and if any are unhappy, are their concerns valid? If so, do whatever is needed to correct that situation. Perhaps most important, have any good possibilities been overlooked?