Notes and Cautions

Go To: Advancing Competitiveness Homepage

Credit: Pixabay

A Conceptual Framework for Economic Decision Making in Advancing Manufacturing Industry Competitiveness: Notes and Cautions

For more information, see NIST AMS 100-80.

There are a number of notes and cautions that should be considered in implementing the framework. For instance, it is important to note that a change agent organization likely needs a diversified portfolio due to diminishing returns, risk of project failure, and avoiding strategic lock in where the organization loses various research capabilities. There is also a need to maintain intellectual humility, as predictions and measures of impact are not absolute. Additionally, the intended purpose of the framework is primarily to ensure that each group or project moves closer to its highest impact potential and not to justify shifting funds from one group to another, which may have unintended consequences. Moreover, selecting the highest-impact projects alone may not necessarily lead to a change agent reaching its highest potential impact, as additional operational factors must be considered. Thus, impact predictions/forecasts are better treated as strategic guides rather than automated lock-in decisions.

Implementing the framework is a substantial challenge, as there are multiple obstacles. One might present the conditions for reaching a change agent’s potential impact as a pyramid. It requires the right culture and incentive structure, standardized methods, tracking and testing, optimizing and researching impact configurations, and implementing the findings that lead to reaching a change agent’s impact potential. This process is itself an exercise in scientific reasoning, presenting challenges that necessitate continued discovery and methodological advancement. Achieving the right conditions is likely difficult and critical, as there can potentially be a misalignment of incentives. 

There are a few assumptions/structures that the framework tends to rely upon. The framework assumes staff have the incentive to report their failures (e.g., errors in prediction) honestly and decision-makers can maintain intellectual humility to accept that their predictions may be wrong and require adjustment. It also assumes the organization is willing to allocate sufficient resources specifically to manufacturing economics and data infrastructure. Without resources and good faith cooperation, the feedback loop breaks. The accuracy of the recalibration factor is entirely dependent on the quality of the error data fed into it.

Ensuring that the feedback loop does not break involves addressing the psychological and structural barriers that prevent honest reporting. If a change agent only rewards high-impact results, researchers may be tempted to over-forecast potential impact and then over-report actual impact. Additionally, staff may fear that reporting a large gap between predicted impact and actual impact will result in reduced funding or negative performance reviews. This effect is similar to that experienced in safety. In settings with elevated safety risk, rewarding zero incidents often leads to the suppression of near miss data, making the environment more dangerous by hiding risks. To ensure the feedback loop functions, the organization will need to shift its value system from outcome-only to accuracy-first:

Reward the "Near Miss" in Forecasting: Just as safety cultures reward the reporting of hazards before they cause injury, an impact culture can reward the identification of prediction errors. An analyst who identifies why a project underperformed provides more value than one who obscures a failure.

Intellectual Humility: The framework assumes that decision-makers maintain humility to accept that their initial predictions might be wrong. This can be formalized by evaluating staff based on the error of their portfolio over time, rather than the success of a single moonshot. There is also a need for humility in impact forecasts: when a project faces high uncertainty or unknown unknowns, intuition may outperform formal impact assessments. This suggests maintaining a secondary pathway for project selection and development that allows intuition-driven opportunities to be pursued alongside data-driven projects. Note that this would likely account for only a minority of projects, which would be highly novel or exploratory in nature.

Decoupling Reporting from Funding: It is crucial that the framework is used to optimize project design rather than to justify shifting funds between groups in a punitive manner, which would immediately trigger defensive (and dishonest) reporting. Stated another way, if funding is coupled with impact predicting and reporting, the power of the framework is diminished.

Leading Indicators as a safety valve: A dual-layered approach with noisy leading indicators, allowing for quicker feedback, and more accurate lagging indicators to serve as the definitive ground truth for periodic structural recalibration can create both timely and more effective assessments. If staff can report a likely failure during the execution phase without penalty, the organization can pivot resources more effectively than waiting for a lagging "actual impact" report.

Framework Components, Logic, and Implementation

Credit: Pixabay The Framework	Credit: Pixabay Framework Logic	Credit: Pixabay Implementation Infrastructure
The Framework's Process	Background: Impact Forecasting	Notes and Cautions
Data Layers and Feed Back Loops	Supporting Logic and Evidence	Standards and Platforms
Metrics and Units of Observation	Economies of Scale	Modular Adoption

Credit: AMS 100-80

Collaboration is a key component to reducing change agent costs and enabling compound learning. If you are considering adopting this framework, consider reaching out to the author Douglas Thomas, Economist: douglas.thomas [at] nist.gov (douglas[dot]thomas[at]nist[dot]gov)