There is no hardcoded limit.
From a practical point of view, I think that group/role membership tend to be unmanageable when there are over a few hundreds. But of course it depends on your management tools and how ofen groups/roles are created, changed...
From a DS point of view, going over a few thousands may decrease performances.
Regards,
Ludovic.