Good question!
The difference between AUI and Atlaskit is the result of the decoupling of Server products and Cloud, mostly due to the fact that the Cloud architecture allows for more rapid development and deployment.
AUI can be considered the legacy implementation (used in Server products). AtlasKit was/is developed together with the Atlassian Design Guide (ADG) version 3 (current version) and should be the most complete implementation of the ADG 3 design guideline. However, AUI has been playing catch-up with ADG and should also match most requirements.
In your Server app, you can use AUI and/or AtlasKit, although it doesn’t seem to be very useful to have both in your project. For cloud add-ons, it is highly recommended to use AtlasKit as the UI will match the host product best.
Keep in mind though, that using React in your server app comes with some caveats, mainly due to the fact that not all Jira versions ship with React. Also, creating the same front-end app for both Server and Cloud versions of the host product is doable, but tricky as they truly are two different beasts.