Parsimony in Scientific Machine Learning

We talked about small data, now let's talk about small models (and code an example)

Thanks for reading! We had a previous series (two parts so far) kick off on how to successfully manage your way in projects where data was limited. To compliment this, let’s talk about models that carry a similar ‘minimalist’ attitude; more simple models, hopefully with explanations when possible, rather than super complex ones. For some problems, the complex ones are the superior choice. But as often as we can, it’s useful to use simple models (at least as a means for comparison and interpretability).

“In philosophy, Occam's razor (also spelled Ockham's razor or Ocham's razor; Latin: novacula Occami) is the problem-solving principle that recommends searching for explanations constructed with the smallest possible set of elements” (source)

In practice, a parsimonious model is one that achieves the desired level of fidelity (prediction, interpretability, …) using as few parameters as possible, avoiding unnecessary complexity (e.g. over fitting). Such models focus only on the most essential features of the data or system, and hence are great at providing insights (again, if only used for a basis of comparison against the final model).

Parsimony is highly valued in modeling and SciML for several reasons: