I’m not sure if I should use CacheFactory. The CacheManager is advertised by official documentation as the preferred way.
How do I cache data in a plugin?
Always depend on CacheManager instead of its super-interface CacheFactory.CacheFactoryexported by Confluence is an instance of TransactionalCacheFactory with often surprising performance characteristics.
I do believe you in your findings. However, not using context:component-scan or context:annotation-config and relying on bean stopped the error from occurring. I would have expected no difference w.r.t the destroy behavior of Spring 
In that case I will try to use the destroyMethod way if the recommendation above is still valid.
Thank you for your help.