Application going down -reaching max DB connections (75) after a plugin installation

Bitbucket Development
,
#1

Hi All,

I have developed a plugin which listens on pull request’s merge, reopen and openRequest events.

After listening to each event, we spawn an executorService thread to do some stuff(some validation and commenting on Pull Request).

After installing this plugin in bitbucket datacenter, the bitbucket application went down in a couple of days with an error saying that “reaching match DB connections(75)”.
On observing the plugin logs, looks like the plugin is trying to access the bitbucket internal DB and the below error is thrown…
“Caused by: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15002ms. at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:603)”

Can anyone tell me why the plugin is trying to access internal DB and how to control the connections?
All I am trying to do is post a simple comment on the Pull Request.

Please tell me if any other information is needed to analyze.

#2

You’ll probably need to post some code, but doing the executorService to spawn other threads is probably the issue. Can you show how you’re doing the commenting (or more importantly getting hold of the services that you’re using to comment)?

#3

Thanks for the reply danielwester

Below is the sample code for PullRequestReopenedEvent type events.

public PullRequestReopenedEventListener(ExecutorService executorService, RepositoryService repositoryService,
                                            PluginSettingsFactory pluginSettingsFactory,
                                            PullRequestService pullRequestService, Logs logs) {
        this.logger = logs.getLogger(this.getClass().getName());
        this.executorService = executorService;
        this.repositoryService = repositoryService;
        this.pluginSettings = pluginSettingsFactory.createGlobalSettings();
        this.pullRequestService = pullRequestService;
    }

    @EventListener
    public void onEvent(final PullRequestReopenedEvent event) {
        logger.info("Pull Request Reopened Event");
        store(event);
    }

    private void store(final PullRequestReopenedEvent event) {
        executorService.execute(new Runnable() {
            @Override
            public void run() {
                submitEventForExecution(event);
            }
        });
    }

    private void submitEventForExecution(final PullRequestReopenedEvent event) {
        try {
            BaseEventListener baseEventListener = new BaseEventListener(repositoryService, pluginSettings, logger, event);
			// get the details maps(repo, title, branch, state.... etc) from Pull Request event details
            Map<String, String> itemsMap = baseEventListener.getRequiredItemsMapFromPlugin();
            if (itemsMap.get("agilestudioEnabler").equals("on") &&
                    itemsMap.get("agilestudioTitleValEnabler").equals("on")) {
				// validating the Pull Request Title. **AgileStudioOperation** is the custom class to do validation.
                AgileStudioOperation agileStudioOperation = new AgileStudioOperation();
                Map<String, Object> agResultMap = agileStudioOperation.isTitleHaveValidAgilestudioItem(
                        itemsMap.get("agilestudioPRURL"), itemsMap.get("prTitle"), itemsMap.get("prToBranch"));
                logger.info("agResultMap: " + agResultMap);
                if (Boolean.valueOf(agResultMap.get("status").toString())) {
                    return;
                }
				// validation failed. Declining the PR with a comment
                if (itemsMap.get("prState") != "DECLINED") {
                    PullRequestDeclineRequest.Builder pullRequestDeclineRequestBuilder = new PullRequestDeclineRequest.Builder(
                            Integer.parseInt(itemsMap.get("toRepoID").trim()),
                            Long.parseLong(itemsMap.get("prID").trim()),
                            Integer.parseInt(itemsMap.get("prVersion").trim()));
                    PullRequestDeclineRequest declineRequest = pullRequestDeclineRequestBuilder.comment(
                            "##### __*" + agResultMap.get("reason").toString() + "*__").build();
                    pullRequestService.decline(declineRequest);
                    logger.info("Declined the Pull Request");
                }
            }
        } catch (Exception e) {
            logger.error("Error in submitting the event for execution" + e);
            logger.error("Exception: ", e);
            throw e;
        }

Likewise, I am doing for PullRequestMergedEvent, PullRequestUpdatedEvent and PullRequestOpenRequestedEvent events also.

#4

My guess is that because you’ve created your own thread and then holding onto pointers to the various service in then - they’re keeping their transactions open and the associated db connections.

So… I would suggest not using executorservice. If you want to do a background service - create an active object entry and then have a scheduled service through Atlassian-scheduler to process that once a minute or so.

Alternatively use ComponentAccessor to dynamically look up the services inside your ExecutorService (and for those that know me - they know that it hurts me suggesting that).

/Daniel

#5

As far as I know, Scheduler jobs are for one time job or cron jobs.

For my plugin, when the event got caught, I trigger some validation and modify the event(ex: decline) immediately. So even if the plugin caught multiple of events at a time, I do the validation and modification parlally with threads.

If I use this scheduler service, all the events that I caught will be served sequentially and eventually time taking to serve for each event will gradually increase. i.e. If I have 1000 events at a particular time and scheduler job time is 1 min, the time taken to serve the 1000th event would be 1000min which is not at all a solution for my requirement.

Please correct me If I am wrong.