Its kind of like considering blindness as someone who can see. It feels like removing a vital part of the human experience to someone who has come to rely on the ability.
Is it really this significant? I don’t think people usually describe it this way. I, for one, really don’t feel like I miss out on a lot.
What I’m attempting (and maybe failing) to say is that it’s a key part of how you perceive the world. To you it isn’t really a big deal, but people who do think this way just view their own thoughts in a fundamentally different fashion, and the idea of such a big difference in that regard is kind of scary or upsetting to think about for some people. I personally think I would be very sad if I suddenly developed aphantasia, even though I don’t think my imagination is as vivid as others.
For me, I wouldn’t equate my ability to visualize things in my head to sight, but maybe hearing or smell. Could I interact with the world without it? Absolutely. But I do a ton with that ability. I hold lists, draw maps, plan routes, visualize models, check the contents of my fridge while at the grocery store. It also helps me make connections between disperate pieces of data. A lot of this I could do with a pencil and paper, but it’s so much faster to pull it up in my head.