The official answer is that you have to submit a request to the billing team to uninstall an app before the cancellation period ends:
Excerpt:
If you would like to delete the subscription immediately and not wait for the cancelation period, please contact the billing team. Upon your request, they can immediately delete the app subscription.
But I agree this is a poor admin experience. It presents possible security issues and definite UX issues.