The user picker method (/rest/api/3/user/picker) doesn’t currently support the ability to filter out JSM customer users (accountType='customer'). There is a feature request that we’d like to measure interest in: a new exclusion filter that operates similarly to the excludeConnectUsers parameter, in that it would exclude JSM customer accounts.
https://ecosystem.atlassian.net/browse/JSDECO-192
If this is of interest to you, please visit the feature request ticket above to vote / comment.