April 28, 2022
I posted this as a Linkedin article - after I started a post and was out of room. After several instances of being temporarily banned from linkedin, I decided to safekeep it here.
Nature is a continuous and multidimensional system. Any model based on a discrete classification will necessarily be a dimension-reducing projection, and will create arbitrary and energy-hungry distortions.
Justin Kunimune, CC BY-SA 4.0, via Wikimedia Commons
This article started as a comment to this post. I exceeded the character limit - so here’s my first ‘article’ here. The top comment on this post explains that “these guidelines are just that: guides. They are not rules or laws.” I disagree.
Sadly institutions are uncomfortable with “Fuzzy logic” and they prefer what we falsely called “precise” rules.
These types of limits are repeatedly (ab)used as such. Institutions rely heavily on processes and regulations based on thresholds and limits. This reliance is deep and overreaching, and it impacts everything these processes “touch”. Similarly, in places with a highly litigious culture, any step outside of professional guideline is a step closer from a potential lawsuit.
Admitting they are not perfect is a first step. But it is not enough. Rather than dismissing the problems they create, we need to honestly and transparently analyse why they are not perfect.
One example: Obstructive Sleep Apnea. Many experts of sleep apnea agree that the AHI (number of apnea and hypopnea per hour) is just a surrogate metric to quantify the severity of a disease that has many facets. Among them the “inventor” of said AHI, the late Prof. Christian Guilleminault. It is obvious that, in the 1980s, when apneas where hand-counted from polysomnographic (PSG) recordings printouts, this metric was a great advancement - efficient and practical enough to help millions of sufferers of the disease who now had access to both diagnostic AND treatment with the CPAP invented by Prof. Collin Sullivan. Since this time though, technological innovations have demonstrated much more advanced capability than the PSG+AHI to measure obstructive sleep apnea and its effects (among them the Sonomat , Watchpat or Sunrise for instance).
Yet, professional bodies in the sleep diagnostic industry will continue to debate around physiological signals threshold-based rules to define AHI and its cut-off values. A 9 sec apnea does not “exist”, but a 11 sec apnea has the same impact on the result than a 30 sec apnea. Some countries/jurisdictions count desaturations from 3%, some from 4%. Then the thresholds to be diagnosed as a mild, medium or severe sufferer will also depend on jurisdictions and will direct whether the patient can benefit or not for reimbursement for the treatment.
There are literally hundred of these variations and distortions (*). Any meta analysis of the existing literature would need to take that into account to actually produce valid results - but there are just too many differences that are not accounted for in these studies, and may not even be reported. From a regulations standpoint, PSG-based AHI is still the “gold standard” or “reference standard” against which innovative technology are expected to prove their worth for the regulators.
Of course, all ML-based AI development in this field will also have to “prove their accuracy” against such an antiquated distorted metric. Non only that, they also being trained on datasets (clinical studies) that are polluted from the start by these distortions as mentioned earlier(*). There is a lot of hype about avoiding bias in AI applications; here is one direction of progress that I think should be studied more.