Great question that has taken a fair bit of debugging.
The short answer is, this happens because the secondary iframe that opens the modal is destroyed when the modal is opened and so we lose access to the callbacks it is holding onto. Unfortunately, there isn’t a quick and easy solution to make it “just work.” I’d recommend moving the responsibility of doing whatever the callback did to the modal itself and letting it perform the callback when its time to cleanup.
The long answer requires explaining a bit about how callbacks work. When you call
t.modal() from the secondary iframe and pass along a function for the
callback, the client library let’s the Trello web client know that there is a callback, and the web client responds and tells the iframe, “hold onto that callback function until I tell you to let it go.” So the secondary iframe effectively keeps a cache of all of the callbacks. This is done because we don’t want to pass an actual function across the wire. When the modal is closed, the Trello web client reaches back out to the iframe and says, “call that function I told you to hold onto and then let it go.”
However, when you open a modal from a secondary iframe, the modal opens and we remove the secondary iframe from the DOM. The web client still sends the message to call the callback and then release it, but because the secondary iframe has been destroyed, the message goes unreceived.
The reason this doesn’t happen when you open a modal directly from a capability (for instance, as the callback for a board button) is because the connectorUrl is a hidden iframe and never destroyed (unless the Power-Up is disabled). Theoretically, we could try to be really clever and hold onto iframes that have callbacks, but this isn’t something we’ve explored at all.
Relatedly, we do manage a popup stack for you if you open a popup from a popup. So the unexpected behavior only occurs when you bring a modal or board bar into the mix.