Yes. There as difference between unfair and unreal; someone who is malnourished when growing up will forever likely be weaker than someone who received a proper sequence of meals.
We should perhaps recognize that and try to compensate for it, and it's not a value judgement on the person so afflicted, but pretending it doesn't exist just confuses matters.
That's been the entire fight over the last 20+ years, does the test identify anything real and if so, what should be done with it (equality of outcomes vs equality of opportunity, e.g.).
We should perhaps recognize that and try to compensate for it, and it's not a value judgement on the person so afflicted, but pretending it doesn't exist just confuses matters.