Happens in most languages.

Also, many languages have a link between deafness and lacking intelligence, e.g. dumb meaning “not able to speak” and “not intelligent”.

In general, being sensitive to people with disabilities (both physical and mental) is a rather young concept, hence anything that would make someone not be able to be part of society is often also an insult.

That’s also why e.g. terms linked deafness/muteness are often an insult to someone’s intelligence, while e.g. terms linked to blindness are not. Blind people might be unable to perform some things seeing people are able to, but blindness doesn’t necessarily limit someone’s ability to be part of a society unaccomodating to people with disabilities.