Customers see a lot of “ServiceProxyDestroyedException” errors after upgrading to Confluence 10.2 when using our apps. They mostly seem to be connected to the Caffeine cache introduced recently.
Has anyone else experienced this?
Any input from Atlassian on this? Maybe @MarekTokarski?
Our apps still worked in Confluence 10.1, and there is not even a mention of the new Caffeine cache in any Release Notes or “Preparing for” pages for Confluence 10.x…
Hi @cheinig
I’m sorry to hear that. It is out of scope of my expertise, unfortunately. I will try to reroute your concern internally.
We are quite surprised that no one else seems to experience similar problems when using caching (especially CacheLoaders) and disabling/enabling the app.
We may have found an ugly workaround that involves holding CacheLoader references in System.getProperties() instead of directly passing them to cacheManager.getCache, so they survive enabling or disabling our apps. We will need to test this approach thoroughly in single- and multi-node environments…
We’re having the same problem 
.
Has Atlassian released any updates regarding this issue yet (@MarekTokarski)?
Why was the new Caffeine cache switched from version 10.1 to 10.2—without mentioning this important change anywhere? Will this change to Caffeine remain in place, or will it be rolled back?
Unfortunately, our workaround has issues that are somewhat expected: when putting non-String objects in System properties, some functions of the System properties break. This seems to lead to Synchrony not starting up.
We really need an update on this, as Customers are increasingly upgrading to Confluence 10.2. In the past, LTS releases did not bring annoying surprises like this…
@cheinig Can you provide a sample stack trace for this? I’m having a difficult time reproducing it, although clearly with the right setup it’s easy to do so.
The old Ehcache-backed implementation did have some unpleasant logic for dealing with these cases, which should have been transferred to the Caffeine-backed impl in 10.2, but clearly something fell between the cracks.
@kmacleod Sure, here’s a sample stack trace for one of the many calls causing the exception:
-- url: /wiki/rest/enterprise-news-bundle/1.0/categories | userName: admin | referer: http://confluence:8090/wiki/plugins/enterprise-news-bundle/news-center/view.action | traceId: 0ab5c13d5e9377e0
org.eclipse.gemini.blueprint.service.importer.ServiceProxyDestroyedException: service proxy has been destroyed
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor$ServiceLookUpCallback.doWithRetry(ServiceDynamicInterceptor.java:101)
at org.eclipse.gemini.blueprint.service.importer.support.internal.support.RetryTemplate.execute(RetryTemplate.java:81)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.lookupService(ServiceDynamicInterceptor.java:427)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:400)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
at jdk.proxy258/jdk.proxy258.$Proxy3635.find(Unknown Source)
at net.seibertmedia.plugin.confluence.bloghighlights.categories.NewsCategoryDaoImpl.getAllNewsCategoriesInternal(NewsCategoryDaoImpl.java:75)
at net.seibertmedia.plugin.confluence.bloghighlights.categories.NewsCategoryDaoImpl.lambda$createCache$0(NewsCategoryDaoImpl.java:69)
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:167)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$0(BoundedLocalCache.java:2707)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2705)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2686)
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:63)
at com.atlassian.confluence.impl.cache.caffeine.CaffeineCacheLoadingImpl.get(CaffeineCacheLoadingImpl.java:31)
at com.atlassian.cache.impl.metrics.InstrumentedCache.get(InstrumentedCache.java:65)
at com.atlassian.cache.nutcluster.asyncinvalidation.AbstractDelegatingCache.get(AbstractDelegatingCache.java:58)
at com.atlassian.cache.nutcluster.asyncinvalidation.AsyncInvalidationCache.get(AsyncInvalidationCache.java:9)
at net.seibertmedia.plugin.confluence.bloghighlights.categories.NewsCategoryDaoImpl.getAllNewsCategories(NewsCategoryDaoImpl.java:86)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.atlassian.activeobjects.tx.TransactionalProxy.invoke(TransactionalProxy.java:57)
at com.atlassian.activeobjects.tx.TransactionalProxy.lambda$executeInTransaction$0(TransactionalProxy.java:47)
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:21)
at com.atlassian.sal.spring.component.SpringHostContextAccessor.lambda$doInTransaction$0(SpringHostContextAccessor.java:72)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at com.atlassian.sal.spring.component.SpringHostContextAccessor.doInTransaction(SpringHostContextAccessor.java:70)
at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:18)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
at jdk.proxy4/jdk.proxy4.$Proxy395.execute(Unknown Source)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:360)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
at jdk.proxy110/jdk.proxy110.$Proxy1509.execute(Unknown Source)
at com.atlassian.activeobjects.internal.SalTransactionManager.inTransaction(SalTransactionManager.java:45)
at com.atlassian.activeobjects.internal.AbstractLoggingTransactionManager.doInTransaction(AbstractLoggingTransactionManager.java:27)
at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.executeInTransaction(EntityManagedActiveObjects.java:227)
at com.atlassian.activeobjects.osgi.ActiveObjectsDelegate.executeInTransaction(ActiveObjectsDelegate.java:327)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:360)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
at jdk.proxy332/jdk.proxy332.$Proxy4422.executeInTransaction(Unknown Source)
at com.atlassian.activeobjects.tx.TransactionalProxy.executeInTransaction(TransactionalProxy.java:45)
at com.atlassian.activeobjects.tx.TransactionalProxy.invokeInTransaction(TransactionalProxy.java:38)
at com.atlassian.activeobjects.tx.TransactionalProxy.invoke(TransactionalProxy.java:30)
at jdk.proxy252/jdk.proxy252.$Proxy3767.getAllNewsCategories(Unknown Source)
at net.seibertmedia.plugin.confluence.bloghighlights.categories.NewsCategoryServiceImpl.getAllNewsCategories(NewsCategoryServiceImpl.java:55)
at net.seibertmedia.plugin.confluence.bloghighlights.categories.NewsCategoryRestResource.getNewsCategories(NewsCategoryRestResource.java:54)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
(...)
If it helps, here are extracts of the involved methods from our code:
@Path("/categories")
@Produces({ MediaType.APPLICATION_JSON })
public class NewsCategoryRestResource {
private final NewsCategoryService newsCategoryService;
// basic DI constructor with @Inject
@GET
@AnonymousSiteAccess
public Response getNewsCategories() {
final List<NewsCategoryDto> newsCategories = newsCategoryService.getAllNewsCategories();
return Response.ok(newsCategories).build();
}
}
@Named
@Slf4j
public class NewsCategoryServiceImpl implements NewsCategoryService {
private final NewsCategoryDao newsCategoryDao;
// basic DI constructor with @Inject
public List<NewsCategoryDto> getAllNewsCategories() {
return newsCategoryDao.getAllNewsCategories();
}
}
@Transactional
public interface NewsCategoryDao {
List<NewsCategoryDto> getAllNewsCategories();
}
@Named
public class NewsCategoryDaoImpl implements NewsCategoryDao, InitializingBean {
private final ActiveObjects ao;
private final CacheManager cacheManager;
private Cache<String, List<NewsCategoryDto>> allCategoriesCache;
// basic DI constructor with @Inject
@Override
public void afterPropertiesSet() {
createCache();
allCategoriesCache.removeAll();
}
private void createCache() {
final CacheSettings cacheSettings = new CacheSettingsBuilder()
.remote().replicateViaInvalidation()
.flushable()
.maxEntries(1) // yeah, could be a CachedReference, but it doesn't matter for the ServiceProxyDestroyedException
.expireAfterAccess(7, TimeUnit.DAYS)
.build();
allCategoriesCache = cacheManager.getCache("CACHE_KEY", key -> getAllNewsCategoriesInternal(), cacheSettings);
}
@Nonnull
private List<NewsCategoryDto> getAllNewsCategoriesInternal() {
return Arrays.stream(ao.find(NewsCategory.class))
.map(NewsCategoryDto::new)
.collect(toList());
}
@Override
public List<NewsCategoryDto> getAllNewsCategories() {
return newArrayList(allCategoriesCache.get("ALL_CATEGORIES_KEY"));
}
}
Thanks @cheinig that’s helpful. The internal handling of CacheLoader when passed as an argument to getCache has always been a little twitchy when it comes to stale OSGi service references. I’ll try and figure out what got missed in the translation from Ehcache to Caffeine, but in the meantime an option is to switch to using the get-with-loader semantics, e.g.
allCategoriesCache = cacheManager.getCache("CACHE_KEY", null, cacheSettings);
allCategoriesCache.get("ALL_CATEGORIES_KEY", key -> getAllNewsCategoriesInternal());
This moves the value lookup into the get call rather than the getCache call. This is usually better, since your plugin’s cache loader object isn’t being retained when your plugin is re-enabled.
Obviously the original form should still be working, and I’ll figure it out and fix it, but this is a workaround in the meantime.
Thanks, great news!
We hope that a fix lands soon, because we (and/or Claude) would have to touch about 100 files to use the get-with-loader semantics.
While you’re at it, you could update https://developer.atlassian.com/server/confluence/atlassian-cache-2-overview/ to mention the “usually better” way.
Any news on the fix? We are reluctant to make changes in 20-30 apps if a Confluence patch release is on the horizon.
Is there a CONFSERVER ticket we (and customers) can follow?
Hi Cheinig, we have made the fix as part of https://jira.atlassian.com/browse/CONFSERVER-101905. It should be out with 10.2.14 for public usage.
Meanwhile, if you want to test the fix out, it will be out in next EAP download. Versions of 11.0 series and greater than 11.0.0-m174 will contain the fix for you to try it out.
Please let us know if you still face any issue.
Thanks