They aren’t concerned you’ve lost too much weight, they are worried about their own self image and what it means that you can actually maintain a healthy weight.
Yup, people call me “skinny” and I’m right at the borderline between normal and overweight. I could lose 20 lbs and still be in the healthy range. In fact, I’m trying to cut about 10 lbs just to make sure I don’t creep up into the overweight range.