That's interesting, nobody has reported this happening on SPARC until now. What does a 'tail -200 /var/dt/Xerrors' show after you turn on encryption and the Sun Rays get stuck?
There's no known workaround other than not turning on encryption. If you want encryption then you need to run SRSS on one of the supported OS releases. Solaris 10 would be the obvious choice.
I don't currently have access to the server, but could it have anything to do with this error that came up during the utconfig:
rm of /var/opt/SUNWbb/root/home/. is not allowed
rm of /var/opt/SUNWbb/root/home/.. is not allowed
bbcpfiles: Warning: security/pam_unix.so.1 not found
ld.so.1: nscd: fatal: libavl.so.1: open failed: No such file or directory
Killed
It came up with this about mid way through but just kept going on afterwards and said it configured successfully. The security/pam_unix stuff sure makes me think it has something to do with it but I don't really know anything. I did do some checking and there is a libavl.so.1 file in the /usr/lib directory but I can't find a pam_unix.so.1 file anywhere.