The term fediverse is used ambiguously. Sometimes it refers to federated services in general but recently it is narrowed down to software that implements the ActivityPub protocol. Matrix however does not implement ActivityPub protocol and is therefore not part of the fediverse. In theory it would be possible for matrix servers to adopt ActivityPub but for the major implementations I don’t see that coming. Alternatively matrix bridges might allow some sort of interop.