As mentioned in the comments, plain text keys aren’t bad because they are necessary. You have to have at least one plain text key in order to be able to use encryption
How does that help when somebody has access to the phone via your PIN or password?
If I’m not mistaken you can save keys in these chips so that they can not be extracted. You can only use the key to encrypt/decrypt/sign/verify by asking the chip to do these operations with your key.
That sounds only marginally better. Access to the phone still means you can create a backup containing the key, so TPM wouldn’t help much.