Hi @SaaketGupta,
There is no need to make the file dynamic as the details are not required when installing the app in a customer site. The Atlassian product will take care of the installation on the selected site and no credentials are needed (the user installing the app will need to be logged in).
When distributing the app to the customer, the customer will upload the app from the product UI directly (see this page) using the baseUrl of the app that your team will provide.
In the same way, if the app is distributed on the Marketplace no credentials.json file is required. The instructions for app developers on how to distribute an app in this way are available here.
I also wrote this some time ago recommending to never commit this file: Use of credentials.json file in a Connect app - #2 by ccurti
Cheers,
Caterina