We all know the drill.
- Atlassian application releases come out fairly quickly, they support every sort of OS and DB combinations.
- As a cross-product of JIRA/Confluence/Bitbucket version * operating system * database * etc., the number of possible combinations quickly grows to several hundreds.
And we want to be sure that we ship great quality add-ons for all the combinations!
Of course, we have automated tests (unit and integration), but we also run ad-hoc or planned manual smoke testing.
In our experience, testing against every point release may not be very important, and luckily OS or DB is rarely a source for bugs, but even with that, you may need to maintain tens of test instances of JIRA and Confluence. Plus you want to do that with pre-populated test data and with sharing among your team members.
How do you guys handle that?
You have a server running every sort of combos that your staff can use? You have VM images? Docker? Vagrant? Puppet? Chef?