Yes, and that is indeed what we did. But what happens is you end up with a backend awkwardly fragmented into restframework viewsets and normal Django views. Often you end up having to maintain both. And then your frontend is in two distinct "worlds" which are developed in completely different ways. Unless you finally get to having an SPA and only JSON on the server side you'll feel forever in limbo. HTMX just makes so much more sense for an existing Django project unless you can afford to stop development and go all in on React, and even then only if it's really necessary to have an SPA.
Yeah - agree that the Django path to add REST endpoints alongside HTML ones is not the best. I like Flask for this reason: it walks the line between HTML rendering and JSON responses really well, IMO.