Critical issue with Active Objects + PostrgreSQL and multiple data schema migrations

active-objects
bitbucket-server
postgres
plugin-development

#1

We develop Awesome Graphs for Bitbucket Server.

We use Active Objects to store our internal index.

Since 2012 till 2018 there several data schema migrations in our app. Recently we released a new version with a new migration.

Our new customers using PostgreSQL after installing the latest version of the app first time are not able to use it and get an error message. There are no such problems with other DB. There were no such problems with PostgreSQL before.

To reproduce the issue you can install our minimal example in Bitbucket 5.15 with PostrgeSQL 9.6. The instance should have at least one repository.

This example illustrates the work of Awesome Graphs. It has a data schema and 5 upgrade tasks. If this example is installed first time, all update tasks are completed, but while trying to access our tables we get an exception (listed in the end of this message). If in this example there are only 4 or less migrations, everything works fine.

But the similar example with 5 migrations for Jira 7.12.3 works fine. The different behaviour may be explained by difference of Active Objects versions:

  • Bitbucket 5.15 uses AO 1.4.0
  • Jira 7.12.3 uses AO 2.0.0

We found a ticket with the same issue. But this ticket doesn’t help to solve the problem.

This is a critical bug! New customers with PostgreSQL are not able to use our app.

What should we do in this situation? What options can we propose to our customers? Can we do any changes in code to resolve this issue?

P. S. Stack trace from our example.

2018-11-01 21:25:23,871 WARN [cop1:thread-1] c.s.bitbucket.cop1.EscalatedExecutor Houston, we have a problem

com.atlassian.activeobjects.internal.ActiveObjectsInitException: bundle [com.stiltsoft.bitbucket.cop1]

at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:95)

at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:86)

at com.atlassian.sal.core.executor.ThreadLocalDelegateCallable.call(ThreadLocalDelegateCallable.java:38)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.lang.Thread.run(Thread.java:748)

… 1 frame trimmed

Caused by: java.lang.RuntimeException: Could not read fields for table AO_BC6212_TASKS

at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:139)

at net.java.ao.schema.ddl.SchemaReader.readFields(SchemaReader.java:126)

at net.java.ao.schema.ddl.SchemaReader.readTable(SchemaReader.java:110)

at net.java.ao.schema.ddl.SchemaReader.access$000(SchemaReader.java:62)

at net.java.ao.schema.ddl.SchemaReader$1.apply(SchemaReader.java:99)

at net.java.ao.schema.ddl.SchemaReader$1.apply(SchemaReader.java:97)

at com.google.common.collect.Iterators$8.transform(Iterators.java:799)

at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)

at com.google.common.collect.Iterators.addAll(Iterators.java:362)

at com.google.common.collect.Lists.newArrayList(Lists.java:160)

at com.google.common.collect.Lists.newArrayList(Lists.java:144)

at net.java.ao.schema.ddl.SchemaReader.readSchema(SchemaReader.java:97)

at net.java.ao.schema.ddl.SchemaReader.readSchema(SchemaReader.java:88)

at net.java.ao.schema.ddl.SchemaReader.readSchema(SchemaReader.java:81)

at net.java.ao.schema.SchemaGenerator.generateImpl(SchemaGenerator.java:107)

at net.java.ao.schema.SchemaGenerator.migrate(SchemaGenerator.java:84)

at net.java.ao.EntityManager.migrateDestructively(EntityManager.java:141)

at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.migrateDestructively(EntityManagedActiveObjects.java:60)

at com.stiltsoft.bitbucket.cop1.upgrade.AbstractActiveObjectsUpgradeTask.upgrade(AbstractActiveObjectsUpgradeTask.java:33)

at com.stiltsoft.bitbucket.cop1.upgrade.UpgradeTaskV2.upgrade(UpgradeTaskV2.java:3)

at com.atlassian.activeobjects.internal.ActiveObjectUpgradeManagerImpl$1.doInTransaction(ActiveObjectUpgradeManagerImpl.java:68)

at com.atlassian.activeobjects.internal.ActiveObjectUpgradeManagerImpl$1.doInTransaction(ActiveObjectUpgradeManagerImpl.java:64)

at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:21)

at com.atlassian.stash.internal.sal.spi.HostContextAccessorImpl.doInTransaction(HostContextAccessorImpl.java:54)

at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:18)

at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)

at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)

at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)

at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)

at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)

at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)

at com.atlassian.activeobjects.internal.SalTransactionManager.inTransaction(SalTransactionManager.java:42)

at com.atlassian.activeobjects.internal.AbstractLoggingTransactionManager.doInTransaction(AbstractLoggingTransactionManager.java:16)

at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.executeInTransaction(EntityManagedActiveObjects.java:204)

at com.atlassian.activeobjects.internal.ActiveObjectUpgradeManagerImpl.upgrade(ActiveObjectUpgradeManagerImpl.java:64)

at com.atlassian.activeobjects.internal.ActiveObjectUpgradeManagerImpl.upgrade(ActiveObjectUpgradeManagerImpl.java:36)

at com.atlassian.activeobjects.internal.AbstractActiveObjectsFactory.upgrade(AbstractActiveObjectsFactory.java:87)

at com.atlassian.activeobjects.internal.AbstractActiveObjectsFactory.create(AbstractActiveObjectsFactory.java:67)

at com.atlassian.activeobjects.internal.DelegatingActiveObjectsFactory.create(DelegatingActiveObjectsFactory.java:32)

at com.atlassian.activeobjects.osgi.TenantAwareActiveObjects$1$1$1.call(TenantAwareActiveObjects.java:91)

… 6 common frames omitted

Caused by: org.postgresql.util.PSQLException: ERROR: cached plan must not change result type

at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2477)

at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2190)

at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:300)

at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428)

at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)

at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169)

at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:117)

at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getResultSetMetaData(DatabaseMetaDataReaderImpl.java:265)

at net.java.ao.schema.helper.DatabaseMetaDataReaderImpl.getFields(DatabaseMetaDataReaderImpl.java:89)

… 45 common frames omitted


#2

We’ve made some tests trying to find roots of the problem.

  1. We tested the hypothesis that the problem is in old version of AO (1.4.0) in Bitbucket 5.15

We took Jira 7.5.0 (using AO 1.4.0) and run the minimal example with 5 migrations for Jira. Everything worked fine.

  1. We noticed, that there are different versions of JDBC drivers
    In Bitbucket 5.15 - postgresql-42.1.4.jar
    In Jira 7.5 - postgresql-9.4.1212.jar

  2. We tested the hypothesis that the problem is in new version of JDBC driver.
    We replaced in Jira 7.5 its driver postgresql-9.4.1212.jar by the driver from Bitbucket 5.15 postgresql-42.1.4.jar and also run the minimal example for Jira. Everything worked fine.

  3. We replaced in Bitbucket 5.15 its driver postgresql-42.1.4.jar by the driver from Jira 7.5 postgresql-9.4.1212.jar and run the minimal example for Bitbucket. Everything worked fine.

  4. We run the minimal example for Bitbucket in Bitbucket 4.14.0. Everything also worked fine.

But in Bitbucket 5.15 our minimal example for Bitbucket fails.
And in real life customers with Bitbucket 5.15 and PostgreSQL are not able to use Awesome Graphs.