I haven't done it with smart cards but I have done it using PDC with iPlanet Certificate Server as my cert server ..
The configuration is a little complicated and there are multiple steps u need to do, see instructions and detailed explanations below ..
The steps below are applicable only for portal sp3 and above ..
There are two parts to cert authentication,
One is configuring the cert auth module and the second is configuring the
gateway to do PDC.
Configuring Cert auth module.
1.) Login to the admin console.
2.) Select Manage Domains.
3.) Select your domain name.
4.) Expand aithentication and then click on Cert.
5.) Match Cert in Ldap ( this value should be checked if the
certificate is stored in the ldap Server).
6.) Match CRL in Ldap ( this value should be checked if the
certificate revocation list is stored in the ldap Server).
7.) URL of the ldap server ( this should be
ldap://servername:portnumber)
8.) Name(DN) of Node to start the search ( this should be the
search criteria for the ldap server, e.g o=Sun Microsystems).
9.) Field in cert to use to access user info in profile (
typically this should be subject CN).
To configure the gateway to complete the authentication , do the
following ( if there are multiple gateways this should be done for all
gateways) :
1.Go to the Admin Console.
2.Select Gateway Management from the menu at the left.
3.Select the Manage Gateway Profile link
4.Click in the text box under the PDC Enabled field and enter the
fully qualified name that is configured for the PDC and any used encoded
devices. This iscase sensitive ( e.g -
toyfactory.Central.Sun.Com).
5.Click the Add button to add this name to the list window.
The following changes below need to be made in the gateway machine ..
6. Go to platform.conf in /etc/opt/SUNWips/platform.conf and
change ips.gateway.trust_all_server_certs=true. Restart the gateway.
7. Launch /install_dir/SUNWips/bin/certadmin and add the root CA
certificates of the cert server using option 3. Restart the gateway.
8. Now login to portal. The user sees a Select Certificate dialog
box and types in a password.
9.The gateway looks at the certificate, checks that the
certificate was issued by a known Certificate Authority, has not expired,
and has not been
tampered with. If the certificate is deemed valid, the gateway
lets the user proceed to the next step in the authentication process.
10.The gateway contacts the PDC authentication module in the server
and passes it the certificate.
The server checks how two attributes, "check CRL" and "check
certstore," have been set.
If "check CRL" has been set to yes, the server checks to see if
the certificate matches an existing certificate revocation list inside
the LDAP server. If
a match exists, the user is allowed to proceed; if not, the
user is denied access.
If "check certstore" has been set to yes, the server attempts
to match the certificate received from the client with an existing
certificate inside the
LDAP server. If a match exists, the user is allowed to proceed;
if not, the user is denied access.
HTH ..