The Data Center documentation teaches plugin developers to avoid hoarding on massive quantities of Hibernate objects in the session. I assume it’s the same for ActiveObjects.
In an upgrade task, I need to perform a check on all my AO objects. I can process them using the streaming API. But the transaction is only committed once the upgrade task is done.
Is it possible to use transactions in ActiveObjectsUpgradeTasks?
Example:
public class V46UpgradeTask implements ActiveObjectsUpgradeTask {
@Override
public void upgrade(ModelVersion currentVersion, ActiveObjects ao) {
List<Integer> listOfIDs = Lists.newArrayList();
ao.stream(AORequirement.class, Query.select().from(AORequirement.class).where("STATUS = ?", AORequirement.STATUS_ACTIVE),
readOnlyAORequirement -> {
if (isNecessaryToUpgrade(readOnlyAORequirement)) {
listOfIDs.add(readOnlyAORequirement.getID());
}
});
for (int i = 0 ; i < listOfIDs.size() ; i += 50) {
List<Integer> batchOf50Items = listOfIDs.subList(i, Math.min(listOfIDs.size(), i + 50));
// Is this really starting a transaction and releasing the objects at the end?
// Probably not, because transactions in Confluence are PROPAGATION_REQUIRED, so we're just propagating the
// parent transaction and achieving nothing.
ao.executeInTransaction(() -> {
AORequirement[] items = ao.find(AORequirement.class, "ID in (" + StringUtils.repeat("?", ", ", batchOf50Items.size()) + ")", batchOf50Items.toArray());
for (AORequirement item : items) {
item.setHtmlflag(1);
item.save();
}
return null;
});
}
Can we have an intervention from an Atlassian developer here?
Even for Hibernate-stored objects documentation doesn’t work: Objects are kept in the Hibernate session even if we’ve used the TransactionTemplate, so the task slows down little-by-little as we go over thousands of pages.
The HibernateSessionManager mentioned by the doc doesn’t commit the transaction, so objects stay in the session according to the very clear documentation about it.
Also, we can’t just import HibernateSessionManager.REQUIRES_NEW_TRANSACTION and start a new TransactionTemplate(REQUIRES_NEW_TRANSACTION), because the OSGi packages aren’t even available.
So it turns out that the plugins can’t flush sessions and commit. When dealing with bulks of objects (e.g. in upgrade tasks for long-running-tasks), the transactions grow bigger and bigger, all the time. It is not possible to flush the transaction entirely (see above).
I’ve looked at the source of a few plugins: None of them does it properly. They all just use transactionTemplate.execute(), which doesn’t flush the session according to the documentation above (and gets slower and slower when you manage 1,000,000 objects).
How to reproduce the issue
This will throw “IllegalStateException: Transaction already active”. Besides, it is dirty to call ContainerManager.get(“hibernateSessionManager”), because we never know when this will be removed.
public String testDoExpensiveOperationsInSeparateTransactions(long pageId) {
Page page = pageManager.getPage(pageId);
String newDate = new Date().toString();
List<Page> pages = page.getChildren();
List<Long> ids = f(pages).map(p -> p.getId());
for (Long child : ids) {
addText(child, "TEXT 1 - " + newDate);
}
return "OK";
}
void addText(long pageId, String html) {
Object object = com.atlassian.spring.container.ContainerManager.getComponent("hibernateSessionManager");
if (!(object instanceof HibernateSessionManager)) {
throw new RuntimeException("ContainerManager.getComponent(\"hibernateSessionManager\") did not return the HibernateSessionManager as usual: " + object);
}
HibernateSessionManager hsm = (HibernateSessionManager) object;
// We start a transaction, and the HSM does .commit(), .flush() and .clear() on the transaction and session.
hsm.executeWithSessionFlushAndClear(Lists.newArrayList(new Object()), 1, 0, o -> {
// We don't need to commit, because the TransactionTemplate will.
Page page = pageManager.getPage(pageId);
pageManager.saveNewVersion(page, new Modification<Page>() {
@Override
public void modify(Page content) {
content.setBodyAsString(content.getBodyAsString() + "<p>" + html + "</p>");
}
});
return null;
});
}
When faced with a similar problem, we took a different route that I mentioned in a past IM (running the task in a separate background thread so that we control everything with regards to the transaction).
I suspect the disconnect between the documentation and the results that you are seeing is that the entire AO task is already wrapped in one single TransactionTemplate#execute() call (the wrapper for which should be visible in a stack trace). This makes adding any additional execute-stuff-inside-a-session business into your code rather pointless because, I think, it just gets slapped onto the end of the parent session.
We did spend some time trying to get this stuff to work, just as you did, but I believe that we gave up once we discovered the separate thread route.
According to my notes, one unexplored avenue was trying to get a reference to the existing hibernate session (created by the parent) and then calling clear() and/or flush() on it, but we never tried this and I don’t know if we’d have run into the same problems with the unavailable OSGi packages that you noted.
I have marked Scott’s approach as a solution, because it is a workaround that gets things done efficiently enough. He’s correct, instead of implementing an upgrade task, we’ve implemented a background job that runs for several days until all issues are migrated. This way, we work around the transactions being incorrectly implemented (or at least the API not being available for developers to perform session flushing and transaction committing).
It is a bad solution and I encourage Atlassian to make sure it is possible for plugin developers to commit and flush big transactions/sessions.