Git push causes bitbucket (via atlas-run) to immediately shut down

Bitbucket Development Bitbucket Server
#1

Has anyone seen this? Bitbucket UI comes up and I can click around for hours. Git fetch is also fine. But the moment I “git push” anything against the built-in “rep_1” repository, Bitbucket immediately shuts down.

There is nothing interesting in the logs and nothing interesting in dmesg. I even ran with all logging set to DEBUG - still nothing.

I’m on Ubuntu 18.04. 24GB of ram on my laptop. No OOM-killer in sight! Disk is 83% used up (37GB available).

ls -alh /usr/bin/atlas-run
lrwxrwxrwx 1 500 500 51 Dec 19 2017 /usr/bin/atlas-run -> /usr/share/atlassian-plugin-sdk-6.3.6/bin/atlas-run

This is happening against versions of “Control Freak” and “Commit Graph” that we know to be fine, and with every version of Bitbucket I’ve tried so far (4.13.0, 5.0.2, 5.5.1, and 6.0.0). I don’t think it’s related to our apps at all.

Normally I’m pretty handy around Bitbucket - for example I just pushed out our newest version of “Commit Graph for Bitbucket Server” (1150 install-count), and I’m proud to say we made a single version of our app that’s compatible from Bitbucket 4.0.0 to 6.0.0. Unfortunately I had to develop against a standard installation of Bitbucket instead of atlas-run to work around this problem. :frowning:

But this “git push” issue causing an immediate shutdown in atlas-run has me completely stumped!

Here’s what the stdout console looks like in atlas-run when it happens (4.13.0 in this example):

[INFO] [talledLocalContainer] 2019-02-17 22:58:15,306 INFO [Caesium-1-2] c.a.b.s.i.i.jobs.StartupChecksJob Running startup jobs for search

[INFO] bitbucket: Shutting down
[INFO] using codehaus cargo v1.4.7
[INFO] [talledLocalContainer] Tomcat 8.x is stopping…
[INFO] [talledLocalContainer] Feb 17, 2019 10:59:56 PM org.apache.catalina.startup.ClassLoaderFactory validateFile

3 Likes
#2

Hi
Same issue. I tried several versions of bitbucket and git

1 Like
#3

Hi Sylvie,

I’ve had a quick attempt at reproducing the problem you describe and was able to push without a problem.

Some questions:

  • Can you please provide the output from “atlas-version”
  • Are you pushing via SSH or HTTP or HTTPS?
  • When did this start occurring? What changes might you have made to your environment since you were able to do this without incident?

I do see your plugin SDK is 6.3.6, which is a bit over a year old. Unrelated, if you are testing/developing plugins for Bitbucket Server 6.0 you will probably want to upgrade to AMPS 8 and plugin SDK 8 so you can test on Java 11.

It would be interesting to know if you can reproduce this outside of a plugin development environment, for example:

cd /tmp
atlas-run-standalone --product bitbucket --version 5.5.1

Regards,
Ben Humphreys
Bitbucket Server

#4

Hi @behumphreys
Let me answer these questions:

ATLAS Version:    6.3.6
ATLAS Home:       /usr/share/atlassian-plugin-sdk-6.3.6
ATLAS Scripts:    /usr/share/atlassian-plugin-sdk-6.3.6/bin
ATLAS Maven Home: /usr/share/atlassian-plugin-sdk-6.3.6/apache-maven-3.2.1
AMPS Version:     6.3.13
--------
Executing: /usr/share/atlassian-plugin-sdk-6.3.6/apache-maven-3.2.1/bin/mvn --version -gs /usr/share/atlassian-plugin-sdk-6.3.6/apache-maven-3.2.1/conf/settings.xml
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-15T00:37:52+06:00)
Maven home: /usr/share/atlassian-plugin-sdk-6.3.6/apache-maven-3.2.1
Java version: 1.8.0_144, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-142-generic", arch: "amd64", family: "unix"

This is for 6.3.6 but I tried 6.3.12 as well. I tried it with 8.0.7 but it lead to another error on build stage (I guess it should be another issue)

I tried it with both ssh and http. Standalone instance started with sdk is affected too. We have a bunch of Bitbucket plugins so this issue causes some problem with development. As for changes made:
I tried to build something for BB 6.0 on Friday and downloaded this dependency. Also upgraded git to 2.20.1 (but I tried to revert it)

#5

Hi Ben
We tested it across our team and seems it can be reproduced on ubuntu only. It runs correctly with other systems like AMI Linux or Mac OSX.

#6

Hi Ben
is there any update on this?
I can reproduce this on 18.04 with Bitbucket 5.14 5.15 and 5.16.

ATLAS Version: 6.3.12
ATLAS Home: /usr/share/atlassian-plugin-sdk-6.3.12
ATLAS Scripts: /usr/share/atlassian-plugin-sdk-6.3.12/bin
ATLAS Maven Home: /usr/share/atlassian-plugin-sdk-6.3.12/apache-maven-3.2.1
AMPS Version: 6.3.21

It also crashed with Atlas-Version 8, but I downgraded as I thought that could be the error.

Best
Chris

#7

I figured out that removing ‘hooks’ folder inside repository (home/shared/data/repositories/1) fixes issue. Not sure if any post processing is working but at least I don’t have to restart Bitbucket every time I push something

1 Like
#8

Just noting that removing the hooks will mean PreRepositoryHooks and PostRepositoryHooks will not be called, which in turns breaks a pretty broad subset of Bitbucket Server (and 3rd-party app) functionality.

#9

Just a quick update for those running into this. I’ve successfully reproduced the behavior internally with atlas-run-standalone. I’ve opened ATLASSDK-164 to track finding the root cause and fixing it.

It does not appear this is a Bitbucket Server issue. Bitbucket Server isn’t crashing; AMPS is shutting it down.

Further investigation is underway. I’d recommend watching the issue for updates.

Best regards,
Bryan Turner
Atlassian Bitbucket

1 Like
#10

To prevent the server from shutting down at git push, you can write input to atlas-run:
yes | atlas-run

I run into the problem with Ubuntu (16.04, 18.04), Bitbucket Server (5.06, 5.15, 5.16), AMPS (6.3.15, 6.3.21), openjdk-8.
A debugging option on line 206 of amps / RunMojo.java would be helpful to find the cause.
I’m writing this comment here because I do not have access to [ATLASSDK-164].

Regards,
Milan

1 Like