Azure Data Factory - API Calls Changed?

Jira Development Jira Cloud
,
1

Hi All,

We have been using ADF to ingest Jira into our data lake for several months now. The connector has worked with no issues and we have been able to pull all issues and all the custom fields, etc from Jira. Our Monday refresh failed due to a schema change in platform API issue and extra API issue. Now all the data is no longer being returned from the extra.api_issue. We are only getting a subset of total issues in Jira.

Is anyone aware of API changes that would break the ADF connector? Is there any other table in the ADF connector that would give me the custom fields? Story points and other values are driven in there for us.

2 Likes
2

Welcome to the Atlassian community @JoseUlloa,

I don’t know enough about how Azure Data Factory works to know what could cause that to break. While there are a number of deprecation notices in the changelog for the Jira Cloud Platform REST APIs in the last couple weeks, those only take effect in the future. The last known breaking change was on 31 May, and that was a change to a delete operation, not get.

Could the problem be caused by a change to Jira configuration by admins, like creating a new issue type or custom fields?

Side note that “ADF” is a bit overloaded. We also use that term to refer to “Atlassian Document Format”.

1 Like
3

I am experiencing the exact same issue as Jose is, and we have been using the Azure Data Factory linked service to JIRA for well over a year now and then last week the extra.api_issue is now pulling significantly less of the data. Can not figure out why at all.

1 Like
4

Welcome @KellyMathison,

Is this the connector you are working with: Free JIRA Azure Data Factory connector

If so, it looks like you might need to post on their forum or check with their technical support.

If not, can you provide more details? It’s not something Atlassian builds or supports but I’d like to help get people having a problem to the people who can help troubleshoot & fix it.

1 Like
5

I am having a similar problem as described above. Trying to pull all issues from all projects returns only 2900 issues. It seems data is being pulled in blocks of 100 issues with consecutive IDs, then it skips some, then 100 more consecutive ones, etc.
Unsure if it’s a problem in JIRA or in the Azure Data Factory connector.

2 Likes
6

Still waiting on getting some support. Now dealing with the back and forth between atlassain and Microsoft. Hopefully someone will figure this out.

1 Like
7

Hi, I’m experiencing the same issue. We’ve been using ADF to ingest Jira into our data lake without problems until now. Our latest refresh also failed due to a schema change in the platform API issue and the extra API issue, resulting in only a subset of total issue being returned from extra.api_issue.
Is there any update or solution available for this problem?
Thanks

1 Like
8

Welcome to the Atlassian developer community @PedroMagdielMaldonad,
And for the rest of the folks on this thread,

There’s clearly a lot of frustration behind these posts. I don’t know much about the Azure Data Factory, but I’m sure it supports some important reporting! Unfortunately, I’m still not clear to which connector you are referring. Nobody has confirmed if I Googled the right one.

This will be fruitless because neither Atlassian nor MS/Azure built the Jira connector. Neither company can support a connector that we don’t know about. You’ll need to refer to the vendor of the connector, who can interface with Atlassian developer support if they need help with our APIs.

Or, if you wanted to “cut out the middleman” of the connector vendor, perhaps Atlassian Analytics is the solution for you.

2 Likes
9

So I have taken a look and this Jira Connector is built into Azure Data Factory, it is offered by Microsoft and it directly calls Jira APIs and returns ‘Tables’ of data not just json output. Not sure if Microsoft has created this connector with the help of Atlassian but something has definitely changed either in APIs from Jira or the connector has broken to do some other change.

Copy data from Jira - Azure Data Factory & Azure Synapse | Microsoft Learn

1 Like
10

@JoseUlloa,

Interesting! I’ve been Atlassian’s “chief partner engineer” (“chief” in the sense of “only”) for many years, and I wasn’t involved. In Atlassian style, I’m sure Azure simply read our API docs and built it.

That said, I’m sure we’d be happy to work with Microsoft if they contact our developer support. As I already indicated, I couldn’t find any potential cause based on the timing you mentioned.

1 Like
12

Same here. Using Synapse with the built-in Jira connector. Worked fine for months. Now I get partial results. No fun.

2 Likes
13

Similar issues here. Also another pipeline that is working, but pulling back a smaller amount of data (12k rows in 9mins vs tens of thousands of rows in 45mins).
Additionally we’re seeing an error “This board does not support sprints”. At first I wondered if an errant ticket had a sprint number/reference assigned to a board that wasn’t set up for sprints, but checking in Jira directly doesn’t show anything untoward.
We’re logging this via MS support and will update if we get any firm answers.

1 Like
14

Would be interested to hear if you get anything back. We are waiting on Azure support to let us know what is happening. The more people bombard them, hopefully they will realize there is an issue.

1 Like
15

Also with this issue dont know the reason but it is also happening to me. Regards,

1 Like
16

I believe would be necessary to know how that Table is composed and why now has less data, is this table created in the Azure side?

1 Like
17

Just to chip in here, that is exactly what I’ve now requested from Atlassian (how the table is composed).

Atlassian support pointed us to Microsoft, Microsoft have pointed us back to Atlassian.

What I don’t think anyone is understanding at the moment is this “Connector” doesn’t seem to have any details as to who “owns” it. So that’s why we’re being passed around. Clearly something has changed for a few of us to be saying we’re having the same issue.

To add clarity around my issue in particular (as this might help others searching!) this is what I sumnmarised and sent to Atlassian support most recently:

We have been in touch with Microsoft and they have confirmed the issue is at source, which is what we previously thought.

The Jira Connector that is used in ADF, as I understand it, is created and maintained by Atlassian, so Microsoft have no way of knowing exactly how it is pulling back data. If this is not the case, please confirm.

To strip this right back to the essential issue:

  1. We are connecting to Jira Cloud using the Jira Connector in ADF
  2. Using the connector we have created a linked service as per Microsoft guide:
    [Copy data from Jira - Azure Data Factory & Azure Synapse | Microsoft Learn|Copy data from Jira - Azure Data Factory & Azure Synapse | Microsoft Learn]
    NB - this linked service is shared with and works without issue on many other pipelines in ADF.
  3. In it’s simplist form, if we then pick a table from the drop down and click “Preview data”, for the most part this returns a very small sample of the data from that table.
  4. Our initial issue was that “Extra”.”Agile_Board_Sprint_Issue” was returning an error:
    ERROR [HY000] [Microsoft][JIRA] (30) Error with HTTP request, response code: 400. Message: [“The board does not support sprints”],{},
  5. Upon further investigation I have found that the following tables also return that error: (there may be others!)
    “Agile”.“Agile_Board_Sprint_Issue”
    “Agile”.“Agile_Board_Sprint_Issue_Fields_Attachment”
    “Agile”.“Agile_Board_Sprint_Issue_Fields_Closed_Sprints”
    “Agile”.“Agile_Board_Sprint_Issue_Fields_Comment_Comments”
    “Agile”.“Agile_Board_Sprint_Issue_Fields_Components”
  6. There are some tables that do return preview data that relate to sprints such as:
    “Agile”.“AgileNoneEpicIssueFieldsClosedSprints”
    “Agile”.“Agile_Board_Backlog_Issues_Fields_Closed_Sprints”
    “Agile”.“Agile_Board_Issue_Fields_Closed_Sprints”
    “Agile”.“Agile_Board_Sprint”

Given the error shown by ADF is “The board does not support sprints” which is a known Atlassian error, I feel like there is some issue in our data somewhere but we’re unsure where it would be.
In the first instance, is it possible to get a list of fields that are in the “Extra”.”Agile_Board_Sprint_Issue” table?

1 Like
18

@AaronHunt,

As if the “back and forth” weren’t frustrating enough, in the back channels of Atlassian support, I’m now the “go to person”. As such, I will continue to “put all the cards on the table”.

I understand that Microsoft have claimed the underlying “connector” is owned by Atlassian. From error messages, I believe they are referring to a library called JIRAODBC, that appears to treat the Jira REST API as a database connection. I am unaware that Atlassian has ever provided such a library. I can find no reference to it via Google, let alone in Atlassian documentation.

From your support case, I have also see the Microsoft documentation about this connector, which makes it clear that Jira Cloud & Jira Data Center are treated the same. That’s already a dangerous assumption since the REST APIs have been diverging for the last 5+ years.

Although I don’t know how that ODBC driver works on the REST API, I do know how the REST APIs work. The driver might be making assumptions about the relationship between Boards & Sprints that are no longer true. Perhaps the driver is relying on how Jira used “out of the box” custom fields to hold the relationships. If it hard coded the custom field references, then I can see how changes in the Jira Cloud REST API could break the driver. However, references to customer field IDs were never recommended for navigating those relationships. That’s why the Jira Software REST API exists as a layer over the Jira Platform REST API. It’s a poor theory because it doesn’t explain why many of you report partial results but it’s the best I can muster without knowing how this ODBC driver works.

I don’t know if any of that helps. I continue to assert that Microsoft should contact our developer support (not customer support). Whether through an ODBC library or the REST API directly, Microsoft depends on us. It doesn’t seem right that should be pushed back to you as customer(s).

19

What I know is that the table always contains 12.400 lines what is suspicious. Everyday counting the same info or same row count seems like some kind of row limit in one side or another. Also it seems the extraction is complete so I would say there has been some kind of row limitation implemented somewhere.

20

This is the issue for us as well. Some of the agile tables are completely broken and the extra API table with all the custom fields does not return all records. Microsoft pointed back to Jira but I am trying to get next level support from them. Maybe someone can take a look at the underlying structure of the connector. Not sure but our reporting have been broken for 3 weeks now.

21

Hi, does anyone has any update about the topic?

I’m still having the same issue than some of you but I do not have further detail from Microsoft neither Jira.

1 Like