Searching for an example of Reporting user personal data

Is there any example to implement the reporting of user personal data following the guidelines User privacy guide for app developers ?

I believe the principle should be the same for all apps , so it will be nice to have an example documented somewhere :

Hi @AtlaBrio ,

Thanks for posting this. Here is the code of a simple Forge example app demonstrating this functionality:

Manifest

modules:
  macro:
    - key: forge-user-privacy-reporting-demo
      function: macro-fn
      title: Forge User Privacy Reporting Demo
      description: Demonstrated how a Forge app should ensure it complies with user privacy guidelines.
  scheduledTrigger:
    - key: personal-data-trigger
      function: pd-trigger-fn
      interval: hour
  function:
    - key: macro-fn
      handler: index.renderMacro
    - key: pd-trigger-fn
      handler: index.reportPersonalDataTrigger
permissions:
  scopes:
    - storage:app
    - report:personal-data
app:
  id: ari:cloud:ecosystem::app/xxxx

index.tsx

import ForgeUI, {
  Button,
  ButtonSet,
  Fragment,
  Heading,
  Macro,
  render,
  Text,
  useAction,
  useState
} from '@forge/ui';
import {
  privacy,
  storage,
  startsWith,
  ListResult,
  Result
} from '@forge/api';

/*
 * This app demonstrates how a Forge app can invoke the personal data reporting API to determine
 * what personal data needs to be purged. The app provides a Confluence macro that allows the
 * user to control the storing of personal data by the app and a control to force the reporting.
 * In addition to providing the user with a manual control for forcing the reporting of personal
 * data, the app also defines scheduled trigger to demonstrate how an app would periodically 
 * report its personal data usage.
 * To ease readability, the app has been kept as simple as possible which results in some
 * limitations. For example, the app would not handle a large number of records with personal data 
 * because it would run into function timeouts. To address this, the reporting would have to be 
 * done in chunks using the async events API:
 *  (https://developer.atlassian.com/platform/forge/runtime-reference/async-events-api/). 
 * 
 * How to use the app:
 * (a) Install the app in Confluence.
 * (b) Insert the app's macro, Forge User Privacy Reporting Demo, in a page.
 * (c) Click the "Store active user" button to store personal details for an active user.
 * (d) Click the "Store closed user" button to store personal details for a closed user.
 * (e) Observe the macro shows two records are being stored.
 * (f) Click the "Report personal data" button and observe the closed account details isremoved from
 *     the app's storage.
 * (g) Click the "Store closed user" button to store personal details for a closed user again.
 * (h) Wait for an hour or so and refresh the page to observe the closed user's account details has
 *     been purged after the app's scheduled trigger runs.
 */

// These account IDs are from https://developer.atlassian.com/cloud/confluence/user-privacy-developer-guide/#testing
const activeTestAccountId = '5be24ad8b1653240376955d2';
const closedTestAccountId = '5be24ba3f91c106033269289';

const accountItemStorageKeyPrefix = 'account-item-';

interface AccountItem {
  accountId: string
  email: string
  updatedAt: string
}

const getStoredAccountItems = async (): Promise<AccountItem[]> => {
  // NOTE: this does not handle arbitrary sizes of data
  const queryResult = await storage.query()
    .where('key', startsWith(accountItemStorageKeyPrefix))
    .getMany() as ListResult<AccountItem>;
  return queryResult.results.map((item: Result<AccountItem>) => {
    return item.value;
  });
}

const deleteAccountItemWithId = async (accountId: string): Promise<void> => {
  const storageKey = `${accountItemStorageKeyPrefix}${accountId}`;
  await storage.delete(storageKey);
}

/**
 * @returns the number of personal data records purged.
 */
const reportPersonalData = async (): Promise<number> => {
  let accountPurgeCount = 0;
  const storedAccountItems = await getStoredAccountItems();
  console.log(`Checking if any personal data needs to be cleaned up...`);
  // NOTE: this does not handle arbitrary sizes of data because the Forge runtime may
  // time out. For a robust implementation, use the Forge Async events API.
  const payload = storedAccountItems.map((accountItem: AccountItem) => {
    return {
      accountId: accountItem.accountId,
      updatedAt: accountItem.updatedAt
    }
  });
  // https://developer.atlassian.com/platform/forge/runtime-reference/privacy-api/
  const updates = await privacy.reportPersonalData(payload);
  if (updates && updates.length) {
    console.log(`Some personal data may need to be purged from the app's data store:`);
    for (const update of updates) {
      if (update.status === 'closed') {
        await deleteAccountItemWithId(update.accountId);
        accountPurgeCount++;
      } else if (update.status === 'updated') {
        // The personal data that the app is storing may be stale. This app won't do 
        // anything in this case, but aniother app may like to delete the record or
        // refresh the data.
      }
      console.log(` * purged ${accountPurgeCount} acount(s).`);
    }
  } else {
    console.log(`No personal data needs to be purged from the app's data store.`);
  }
  return accountPurgeCount;
}

export const reportPersonalDataTrigger = async (): Promise<any> => {
  const accountPurgeCount = await reportPersonalData();
  const triggerResponse = {
    body: `{"records-purged": ${accountPurgeCount}}`,
    headers: {
      'Content-Type': ['application/json'],
      'X-Request-Id': [`rnd-${Math.random() }`]
    },
    statusCode: 200,
    statusText: 'OK'
  };
  return triggerResponse;
}

const App = () => {

  const [initialStoredAccountItems] = useAction(value => value, async () => await getStoredAccountItems());
  const [storedAccountItems, setStoredAccountItems] = useState<AccountItem[]>(initialStoredAccountItems);

  const storeAccountItem = async (accountItem: AccountItem): Promise<void> => {
    const storageKey = `${accountItemStorageKeyPrefix}${accountItem.accountId}`;
    await storage.set(storageKey, accountItem);
    const storedAccountItems = await getStoredAccountItems();
    setStoredAccountItems(storedAccountItems);
  }

  const onStoreActiveAccountItem = async (): Promise<void> => {
    const activeAccountItem: AccountItem = {
      accountId: activeTestAccountId,
      email: 'active-user@example-domain.com',
      updatedAt: new Date().toISOString()
    }
    await storeAccountItem(activeAccountItem);
  }

  const onStoreClosedAccountItem = async (): Promise<void> => {
    const closedAccountItem: AccountItem = {
      accountId: closedTestAccountId,
      email: 'closed-user@example-domain.com',
      updatedAt: new Date().toISOString()
    }
    await storeAccountItem(closedAccountItem);
  }

  const onReportPersonalData = async () => {
    const accountPurgeCount = await reportPersonalData();
    if (accountPurgeCount) {
      const storedAccountItems = await getStoredAccountItems();
      setStoredAccountItems(storedAccountItems);
    }
  }

  const renderStoredAccountItems = () => {
    if (storedAccountItems.length) {
      return storedAccountItems.map((accountItem: AccountItem) => {
        return (
          <Text>* {accountItem.accountId}: {accountItem.email} (last updated at {accountItem.updatedAt})</Text>
        );
      })
    } else {
      return (<Text>There is not any stored account data</Text>);
    }
  }

  return (
    <Fragment>
      <Heading size="large">Forge user privacy reporting demo</Heading>

      <Heading size="medium">Account data</Heading>
      {renderStoredAccountItems()}

      <Heading size="medium">Account operations</Heading>
      <ButtonSet>
        <Button text="Store active user" onClick={onStoreActiveAccountItem} />
        <Button text="Store closed user" onClick={onStoreClosedAccountItem} />
        <Button text="Report personal data" onClick={onReportPersonalData} />
      </ButtonSet>
    </Fragment>
  );
};

export const renderMacro = render(
  <Macro
    app={<App />}
  />
);

I’ve included comments in the example app to explain certain aspects.

Please let me know if this is the sort of thing you are looking for. If it is, I can share this along side the other Forge example apps.

Regards,
Dugald

1 Like

Hi @dmorrow , thank you for the feedback.
I think the pagination is missing in this example, since the forge storage cannot return more than 20 records, and the privacy API cannot take more than 90 accountIds.
While trying on my own, I discover that the usage of scope ‘report:personal-data’ was missing in page https://developer.atlassian.com/platform/forge/runtime-reference/privacy-api/

Overall, it will great if this privacy doc page could be completed with one example and the scope

2 Likes

Hi @AtlaBrio ,

With this example, I’ve tried to find the right balance between simplicity and comprehensiveness. As explained in the code, it wouldn’t handle large amounts of records, but I feel it demonstrated the key aspects relating to the reporting of the apps storage of personal data.

I’ve created an internal task to update the reference documentation as you suggest.

Regards,
Dugald

1 Like

Sorry @dmorrow , i didnt see the comments in the code. Thanks for the tip for the async events ! completely forgot 25s time limit.
I agree it will be nice to have the example in other Forge example apps. also, just a personal opinion, but it will better if it could include the async events for large amount of data, because at the end, it’s the same for all apps, developers will just have to copy paste the example and only replace the methods to get accountIds and to delete
Thanks a lot for the request on the privacy page

2 Likes

There has already been a Forge GDPR Polling Example app within the Ecosystem project (rather than the Forge examples project as usual). It predates the Async Events API though and might need some refactoring accordingly (IIRC it uses a custom queue of sorts, but implements pagination already) - maybe you could evolve and/or add that one to the Forge example app section(s) @dmorrow?

1 Like

Hi @sopel and @AtlaBrio ,

Thanks for your feedback. Here’s a new version of the app that includes processing records in an asynchronous, paginated manner.

manifwest.yml

modules:
  macro:
    - key: forge-user-privacy-reporting-demo
      function: macro-fn
      title: Forge User Privacy Reporting Demo
      description: Demonstrated how a Forge app should ensure it complies with user privacy guidelines.
  consumer:
    - key: report-pd-consumer
      queue: report-pd-queue
      resolver:
        function: on-report-pd-fn
        method: pd-report-listener
  scheduledTrigger:
    - key: personal-data-trigger
      function: pd-trigger-fn
      interval: hour
  function:
    - key: macro-fn
      handler: index.renderMacro
    - key: on-report-pd-fn
      handler: index.onReportPDQueueData
    - key: pd-trigger-fn
      handler: index.reportPersonalDataTrigger
permissions:
  scopes:
    - storage:app
    - report:personal-data
app:
  id: ari:cloud:ecosystem::app/xxxxxxx

index.tsx

import ForgeUI, {
  Button,
  ButtonSet,
  Fragment,
  Heading,
  Macro,
  render,
  Text,
  useAction,
  useState
} from '@forge/ui';
import {
  privacy,
  storage,
  startsWith,
  ListResult,
  Result
} from '@forge/api';
import { Queue } from '@forge/events';
import Resolver from "@forge/resolver";

/*
 * This app demonstrates how a Forge app can invoke the personal data reporting API to determine
 * what personal data needs to be purged. The app provides a Confluence macro that allows the
 * user to control the storing of personal data by the app and a button within the macro to force 
 * the reporting for testing purposes.
 * In addition to providing the user with a manual control for forcing the reporting of personal
 * data, the app also defines a scheduled trigger to demonstrate how an app would periodically 
 * report its personal data usage. The scheduled trigger fires once per how which is likely more
 * often than a real app would need to do.
 * 
 * How to use the app:
 * (a) Install the app in Confluence.
 * (b) Insert the app's macro, Forge User Privacy Reporting Demo, in a page.
 * (c) Click the "Store active user" button to store personal details for an active user.
 * (d) Click the "Store closed user" button to store personal details for a closed user.
 * (e) Observe the macro shows two records are being stored.
 * (f) Click the "Report personal data" button and observe the closed account details isremoved from
 *     the app's storage.
 * (g) Click the "Store closed user" button to store personal details for a closed user again.
 * (h) Wait for an hour or so and refresh the page to observe the closed user's account details has
 *     been purged after the app's scheduled trigger runs.
 */

// These account IDs are from https://developer.atlassian.com/cloud/confluence/user-privacy-developer-guide/#testing
const activeTestAccountId = '5be24ad8b1653240376955d2';
const closedTestAccountId = '5be24ba3f91c106033269289';

const accountRecordStorageKeyPrefix = 'account-item';

// This constant is from https://developer.atlassian.com/platform/forge/runtime-reference/privacy-api/
const maxAllowedRecordsToReportPerBatch = 90;

const oneDayInMillis = 24 * 60 * 60 * 1000;

// This constant defines how often the reporting will be done per account record.
// const personalDataReportingPeriodMillis = 7 * oneDayInMillis; // A realisitc value for an app to use.
const personalDataReportingPeriodMillis = 0; // A value to use for testing the app.

// This constant defines the maximum number of records it will attempt to report on per batch. If there
// are additional records to report on, then a request will be enqueued using the Async events API.
const maxRecordsToReportPerBatch = 1;

// The UI is not the primary focus of this example app, but it does provide rudimentary presentation
// of all the data in set sizes defined by this constant.
const maxRecordsForUI = 10;

interface AccountRecord {
  storageKey: string
  accountId: string
  email: string
  updatedAt: string
  nextReportDueBy: string
}

interface AccountRecordsQueryResult {
  accountRecords: AccountRecord[]
  nextCursor: undefined | string
}

interface AccountReportingAsyncContext {
  nextCursor: undefined | string
}

interface PersonalDataReportItem {
  accountId: string
  updatedAt: string
}

const pdReportingQueue = new Queue({ key: 'report-pd-queue' });
const pdReportingResolver = new Resolver();

const buildRecordStorageKey = (accountId: string): string => {
  return `${accountRecordStorageKeyPrefix}-${accountId}`;
}

const getStoredAccountRecords = async (maxRecords: number, cursor: undefined | string): Promise<AccountRecordsQueryResult> => {
  console.log(`Querying stored account recors with cursor ${cursor}...`);
  const limit = Math.min(maxAllowedRecordsToReportPerBatch, maxRecords);
  const queryResult = await storage.query()
    .where('key', startsWith(accountRecordStorageKeyPrefix))
    .limit(limit)
    .cursor(cursor)
    .getMany() as ListResult<AccountRecord>;
  const nextCursor = queryResult.nextCursor;
  const accountRecords = queryResult.results.map((accountRecordResult: Result<AccountRecord>) => {
    return accountRecordResult.value;
  });
  const accountRecordsQueryResult: AccountRecordsQueryResult = {
    accountRecords: accountRecords,
    nextCursor: nextCursor
  }
  return accountRecordsQueryResult;
}

const storeAccountRecord = async (accountId: string, emailAddressPrefix: string): Promise<void> => {
  const storageKey = `${buildRecordStorageKey(accountId)}`;
  const now = new Date();
  const nextReportDueTime = new Date(now.getTime() + personalDataReportingPeriodMillis);
  const accountRecord: AccountRecord = {
    storageKey: storageKey,
    accountId: accountId,
    email: `${emailAddressPrefix}@example-domain.com`,
    updatedAt: now.toISOString(),
    nextReportDueBy: nextReportDueTime.toISOString()
  }
  await storage.set(storageKey, accountRecord);
}

const deleteAccountRecordsWithId = async (accountId: string): Promise<void> => {
  const accountStoragePrefix = buildRecordStorageKey(accountId);
  const queryResult = await storage.query()
    .where('key', startsWith(accountStoragePrefix))
    .getMany() as ListResult<AccountRecord>;
  for (const accountRecordResult of queryResult.results) {
    console.log(` * deleting record ${accountRecordResult.value.storageKey}`);
    await storage.delete(accountRecordResult.value.storageKey);
  }
}

/**
 * @returns true if the reporting is complete, false otherwise.
 */
const reportPersonalDataAsync = async (cursor: undefined | string): Promise<boolean> => {
  let accountPurgeCount = 0;
  const accountRecordsQueryResult = await getStoredAccountRecords(maxRecordsToReportPerBatch, cursor);
  console.log(`Checking if any personal data needs to be cleaned up...`);
  // NOTE: this does not handle arbitrary sizes of data because the Forge runtime may
  // time out. For a robust implementation, use the Forge Async events API.
  const payload: PersonalDataReportItem[] = [];
  for (const accountRecord of accountRecordsQueryResult.accountRecords) {
    console.log(`Analysing record ${JSON.stringify(accountRecord, null, 2)}...`);
    const now = new Date().getTime();
    const isReportDue = accountRecord.nextReportDueBy ? now > Date.parse(accountRecord.nextReportDueBy) : true;
    if (isReportDue) {
      const item: PersonalDataReportItem = {
        accountId: accountRecord.accountId,
        updatedAt: accountRecord.updatedAt
      }
      payload.push(item);
    } else {
      console.log(`Skipping the reporting of account ${accountRecord.accountId} since a report is not due until ${accountRecord.nextReportDueBy}.`);
    }
  }
  console.log(`Built payload: ${JSON.stringify(payload, null, 2)}`);
  // https://developer.atlassian.com/platform/forge/runtime-reference/privacy-api/
  const updates = await privacy.reportPersonalData(payload);
  if (updates && updates.length) {
    console.log(`Some personal data may need to be purged from the app's data store:`);
    for (const update of updates) {
      if (update.status === 'closed') {
        await deleteAccountRecordsWithId(update.accountId);
        accountPurgeCount++;
      } else if (update.status === 'updated') {
        // The personal data that the app is storing may be stale. This app won't do 
        // anything in this case, but aniother app may like to delete the record or
        // refresh the data.
      }
      console.log(` * purged ${accountPurgeCount} acount(s).`);
    }
  } else {
    console.log(`No personal data needs to be purged from the app's data store for query with cursor ${cursor}.`);
  }

  const couldBeMoreRecordsToReportOn = accountRecordsQueryResult.nextCursor !== undefined;
  if (couldBeMoreRecordsToReportOn) {
    console.log(`Pushing PD reporting queue item with cursor ${accountRecordsQueryResult.nextCursor}...`);
    const accountReportingAsyncContext: AccountReportingAsyncContext = {
      nextCursor: accountRecordsQueryResult.nextCursor
    }
    await pdReportingQueue.push({
      "accountReportingAsyncContext": accountReportingAsyncContext as any
    });
  }
  const reportingComplete = !couldBeMoreRecordsToReportOn;
  return reportingComplete;
}

pdReportingResolver.define("pd-report-listener", async (queueItem) => {
  const accountReportingAsyncContext = queueItem.payload.accountReportingAsyncContext as AccountReportingAsyncContext;
  console.log(`Received PD reporting queue item with cursor ${accountReportingAsyncContext.nextCursor}`);
  await reportPersonalDataAsync(accountReportingAsyncContext.nextCursor);
});
export const onReportPDQueueData = pdReportingResolver.getDefinitions();

export const reportPersonalDataTrigger = async (): Promise<any> => {
  const cursor = undefined;
  const reportingComplete = await reportPersonalDataAsync(cursor);
  const triggerResponse = {
    body: `{"reportingComplete": ${reportingComplete}}`,
    headers: {
      'Content-Type': ['application/json'],
      'X-Request-Id': [`rnd-${Math.random() }`]
    },
    statusCode: 200,
    statusText: 'OK'
  };
  return triggerResponse;
}

const MacroApp = () => {

  const [initialStoredAccountRecords] = useAction(value => value, async () => await getStoredAccountRecords(maxRecordsForUI, undefined));
  const [accountRecordsQueryResult, setAccountRecordsQueryResult] = useState<AccountRecordsQueryResult>(initialStoredAccountRecords);

  const onRefreshView = async (): Promise<void> => {
    const cursor = undefined;
    const accountRecordsQueryResult = await getStoredAccountRecords(maxRecordsForUI, cursor);
    setAccountRecordsQueryResult(accountRecordsQueryResult);
  }

  const onViewNextSetOfRecords = async (): Promise<void> => {
    const cursor = accountRecordsQueryResult.nextCursor;
    const nextAccountRecordsQueryResult = await getStoredAccountRecords(maxRecordsForUI, cursor);
    setAccountRecordsQueryResult(nextAccountRecordsQueryResult);
  }

  const onDeleteAccountRecords = async (): Promise<void> => {
    await deleteAccountRecordsWithId(activeTestAccountId);
    await deleteAccountRecordsWithId(closedTestAccountId);
    await onRefreshView();
  }

  const onStoreActiveAccountRecords = async (): Promise<void> => {
    await storeAccountRecord(activeTestAccountId, 'active-user');
    await onRefreshView();
  }

  const onStoreClosedAccountRecords = async (): Promise<void> => {
    await storeAccountRecord(closedTestAccountId, 'closed-user');
    await onRefreshView();
  }

  const onReportPersonalData = async () => {
    const cursor = undefined;
    await reportPersonalDataAsync(cursor);
  }

  const renderStoredAccountRecords = () => {
    if (accountRecordsQueryResult.accountRecords.length) {
      return accountRecordsQueryResult.accountRecords.map((accountRecord: AccountRecord) => {
        return (
          <Text>* {accountRecord.accountId}: {accountRecord.email}</Text>
        );
      })
    } else {
      return (<Text>There is not any stored account data</Text>);
    }
  }

  return (
    <Fragment>
      <Heading size="large">Forge user privacy reporting demo</Heading>

      <Heading size="medium">Account data</Heading>
      {renderStoredAccountRecords()}

      <Heading size="medium">View operations</Heading>
      <ButtonSet>
        <Button text="Refresh" onClick={onRefreshView} />
        <Button text="View next set of records" disabled={accountRecordsQueryResult.nextCursor === undefined} onClick={onViewNextSetOfRecords} />
      </ButtonSet>

      <Heading size="medium">Account operations</Heading>
      <ButtonSet>
        <Button text="Delete users" disabled={accountRecordsQueryResult.accountRecords.length === 0} onClick={onDeleteAccountRecords} />
        <Button text="Store active users" onClick={onStoreActiveAccountRecords} />
        <Button text="Store closed users" onClick={onStoreClosedAccountRecords} />
        <Button text="Report personal data" onClick={onReportPersonalData} />
      </ButtonSet>
    </Fragment>
  );
};

export const renderMacro = render(
  <Macro
    app={<MacroApp />}
  />
);

Please let me know if you spot any issues or if it is omitting important aspects relating to the reporting of personal data.

I’ve also included one or two tweaks such as avoiding over reporting.

Regards,
Dugald

2 Likes