EDIT (8-Sep-2022): I’m not sure this solution works as I believe the ADMIN scope is still required.
I assume your use of the term plugin refers to a Connect app you have developed.
Instead of adding the ADMIN scope to your app, you could add the ACT_AS_USER scope and make a user impersonated API call from your app’s backend. Obviously you’d need to know that the user you’re impersonating has admin permissions, so you may be able to employ the user_is_admincondition so that the relevant functionality is only available to admins.