An article in HBR, April 2013 by Robert C. Merton - the co-inventor of the famous option theory. Therefore I wrote about it in UnRisk Insight - When Three Rights Make A Wrong.
But this is not only essential for finance - it has an effect on mathematical problem solving in general. All innovations involve trade-offs between risk and benefits. In mathematical problem solving risks are intrinsic in the model and in how it is used. But how it may be used is difficult to model.
But first, it is fundamental to distinguish between a model and its representation. In computational mathematics the representations are usually programs - and they really matter.
Take a process in a blast furnace - if you want to improve automation you need to model various phenomena, like solid flows of iron ore and coke, liquid due to melting, gas from bottom to the top, energy - heat conduction and convection and chemical reactions. The above picture shoes the models of the solid flows.
The typical size of the problem: 40.000 unknowns per scalar components of 20-30 unknown functions (temperature, max concentrations, velocities, ..). The model is a complex system of PDEs of the convection-reaction-diffusion type and its representation finite elements. But if you do not know a few "dirty" tricks, like streamline diffusion, the complex model information gets lost in the numerical jungle.
We did it right and we managed to reduce the computing time to 4hours for a real-time day. And this is also essential.
Merton writes: a more-complete but more-complicated model may carry greater risk than a cruder one if the user is not qualified for the job. This is true in principle, but especially when users are human. In automation "users" are technical systems.
Organize your system orthogonally
However, we highly recommend to organize problem-models-methods orthogonally - and be careful with automated algorithm selection (if models are quite complicated).
In finance the first defense front of risk management is model validation - in short, portfolio across model scenario calculations, then portfolio across all types of risk scenarios.
You may have chosen the right model for the problem, the right representation, curated the date, ... but if something little in your solvers or calibration schemes is not adequate, you might better apply a tea-leave-reading approach (especially when inverse problems are involved).
An adequate assessment of the risks involved in innovation requires a careful analysis of (model-method) consequences. It might be by a mathematical model.