I had a REST Endpoint with a dialog that worked on script runner verion 5.0.11 .
We upgrade the script runner to version 5.4.39 and now, The dialog opens in a new window.
In the new version:
In the old version it looks like a regular jira dialog.
I use this code: