Welcome to the Atlassian developer community @NickStefan,
I think the first thing to understand is that Confluence Cloud and Server/DC are diverging products and it would be better to threat each differently. I realize that’s double the work, but the answers to most of your questions are different for each. For example, you’ve found the Connect App possibility but that would only work for Confluence Cloud.
It’s not a bug. The REST API respects the permissions of the user who is authorized in the REST API. There are many API clients that depend on this behavior.
No. That’s the intended behavior.
No. Undocumented APIs are not supported. Use only at your own risk and that of your customers.
Backup and restore is a special case. While I am aware many backup/restore solutions rely on REST APIs, this can be a risky model in general and for Confluence (and Jira) specifically because the REST APIs do not guarantee an atomic nor complete snapshot. And Confluence has some configuration sensitivity, whereby the page content might not restore correctly without first restore configuration.
That’s why Atlassian recommendations for on-premise backup have both filesystem-level and database-level operations. And why the link above indicates exploration of how vendors would want to access complete and atomic snapshots of data, when some of our customers have GBs of data.
While I take your point about the inconsistency between what an admin can do in the UI vs in the REST API, I think the “bug” is not so much that REST APIs cannot be used to access private content, but that appropriate APIs do not exist for admins to automate backup. Again, that is a gap @VamsiBhagi is trying to fill so please do engage on the above linked post.