Unable to update from proxy server

I just recently installed Sol10 on 12 Sun workstations. All of these machines were able to successfully update themselves by using the SUC proxy server. I configured and updated them using smpatch. By default, the SUC packages are installed as one of these patches. After installing the patches and rebooting the machines, these machines are no longer able to get updates from the proxy server. I now get the following error:

Error: Unable to download entitlement information using the update server proxy.

Cannot connect to retrieve : Server returned HTTP response code: 500 for URL: http://xxxxx.onu.edu:3816/solaris/

Message was edited by: ForumModerator

[682 byte] By [ohionorthern] at [2007-11-26 6:03:24]
# 1

I suspect more information on this issue could be gathered from the log files in /var/patchsvr/log/localhost_log.<date>.txt

on the system which is running the Sun Update Connection Proxy.

If you could please post the relevant errors or stack traces you see at the time of failure it would be most helpful in solving this problem.

TIA

-ethan

ejrider at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 2

2005-08-04 11:25:20 Thu Aug 04 11:25:20 EDT 2005(DEBUG) => com.sun.patchpro.server.PatchProServerServlet@faa550 <=UserID from Auth header_0

2005-08-04 11:25:20 Thu Aug 04 11:25:20 EDT 2005(INFO) => com.sun.patchpro.server.PatchProServerServlet@faa550 <=Protocol version: 2.1

2005-08-04 11:25:20 Thu Aug 04 11:25:20 EDT 2005(DEBUG) => com.sun.patchpro.server.ServerPatchServiceProvider@1e903d5 <=cacheFile is: /var/sadm/spool/patchsvr/Database/https%3A%2F%2Fgetupdates.sun.com%2Fsolaris%2F %2FDatabase%2Fcurrent.zip

2005-08-04 11:25:20 Thu Aug 04 11:25:20 EDT 2005(DEBUG) => com.sun.patchpro.server.ServerPatchServiceProvider@1e903d5 <=POST String: action=getFile&name=Database%2Fcurrent.zip&version=2.1

2005-08-04 11:25:20 Thu Aug 04 11:25:20 EDT 2005(DEBUG) => com.sun.patchpro.server.PatchProServerServlet@faa550 <=PatchProServerServlet getDocumentation/getFile done. Status: 200

2005-08-04 11:25:50 Thu Aug 04 11:25:50 EDT 2005(DEBUG) => com.sun.patchpro.server.PatchProServerServlet@faa550 <=UserID from Auth header_0

2005-08-04 11:25:50 Thu Aug 04 11:25:50 EDT 2005(INFO) => com.sun.patchpro.server.PatchProServerServlet@faa550 <=Protocol version: 2.1

2005-08-04 11:25:50 Thu Aug 04 11:25:50 EDT 2005(DEBUG) => com.sun.patchpro.server.ServerPatchServiceProvider@1e903d5 <=cacheFile is: /var/sadm/spool/patchsvr/entitlement/https%3A%2F%2Fgetupdates.sun.com%2Fsolaris %2F%2Fentitlement_lps

2005-08-04 11:25:50 Thu Aug 04 11:25:50 EDT 2005(DEBUG) => com.sun.patchpro.server.ServerPatchServiceProvider@1e903d5 <=POST String: action=getEntitlement&name=&version=2.1

2005-08-04 11:25:50 Thu Aug 04 11:25:50 EDT 2005(ERROR) => com.sun.patchpro.server.PatchProServerServlet@faa550 <=Problem detected while servicing "downloadEntitlementFile" request. Cannot connect to retrieve : java.security.ProviderException: Could not obtain session

2005-08-04 11:25:50 Thu Aug 04 11:25:50 EDT 2005(WARNING) => com.sun.patchpro.util.LocalizedMessages@6963d0 <=Corrupt message catalog: locale="en" messageID="Cannot connect to retrieve : java.security.ProviderException: Could not obtain session" default english="Cannot connect to retrieve : java.security.ProviderException: Could not obtain session"

2005-08-04 11:25:50 Thu Aug 04 11:25:50 EDT 2005(ERROR) => com.sun.patchpro.server.PatchProServerServlet@faa550 <=com.sun.patchpro.util.CannotConnectException: Cannot connect to retrieve : java.security.ProviderException: Could not obtain session

at com.sun.patchpro.util.CachingDownloader.establishConnection(CachingDownloader.j ava:612)

at com.sun.patchpro.util.CachingDownloader.setSourceURL(CachingDownloader.java:274 )

at com.sun.patchpro.util.CachingDownloader.setupCache(CachingDownloader.java:200)

at com.sun.patchpro.util.CachingDownloader.<init>(CachingDownloader.java:185 )

at com.sun.patchpro.server.ServerPatchServiceProvider.downloadFile(ServerPatchServ iceProvider.java:775)

at com.sun.patchpro.server.ServerPatchServiceProvider.downloadEntitlementFile(Serv erPatchServiceProvider.java:847)

at com.sun.patchpro.server.PatchServerProxy.downloadEntitlementFile(PatchServerPro xy.java:145)

at com.sun.patchpro.server.PatchProServerServlet.doPost(PatchProServerServlet.java :1330)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFil terChain.java:247)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain .java:193)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java: 243)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:943)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java: 190)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)

at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:943)

at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2347)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)

at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.jav a:170)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:170)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:468)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:943)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:17 4)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:943)

at org.apache.catalina.connector.http.HttpProcessor.process(HttpProcessor.java:102 7)

at org.apache.catalina.connector.http.HttpProcessor.run(HttpProcessor.java:1125)

at java.lang.Thread.run(Thread.java:595)

Caused by:

javax.net.ssl.SSLException: java.security.ProviderException: Could not obtain session

at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:166)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1476)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1443)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1 426)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:10 45)

at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:405)

at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractD elegateHttpsURLConnection.java:170)

at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.j ava:828)

at com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl.getOutput Stream(HttpsURLConnectionOldImpl.java:200)

at com.sun.patchpro.util.Downloader.connectToURL(Downloader.java:343)

at com.sun.patchpro.util.CachingDownloader.establishConnection(CachingDownloader.j ava:584)

at com.sun.patchpro.util.CachingDownloader.setSourceURL(CachingDownloader.java:274 )

at com.sun.patchpro.util.CachingDownloader.setupCache(CachingDownloader.java:200)

at com.sun.patchpro.util.CachingDownloader.<init>(CachingDownloader.java:185 )

at com.sun.patchpro.server.ServerPatchServiceProvider.downloadFile(ServerPatchServ iceProvider.java:775)

at com.sun.patchpro.server.ServerPatchServiceProvider.downloadEntitlementFile(Serv erPatchServiceProvider.java:847)

at com.sun.patchpro.server.PatchServerProxy.downloadEntitlementFile(PatchServerPro xy.java:145)

at com.sun.patchpro.server.PatchProServerServlet.doPost(PatchProServerServlet.java :1330)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFil terChain.java:247)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain .java:193)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java: 243)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:943)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java: 190)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)

at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:943)

at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2347)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)

at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.jav a:170)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:170)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:468)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:943)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:17 4)

at org.apache.catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)

at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:943)

at org.apache.catalina.connector.http.HttpProcessor.process(HttpProcessor.java:102 7)

at org.apache.catalina.connector.http.HttpProcessor.run(HttpProcessor.java:1125)

at java.lang.Thread.run(Thread.java:595)

Caused by: java.security.ProviderException: Could not obtain session

at sun.security.pkcs11.SessionManager.getOpSession(SessionManager.java:122)

at sun.security.pkcs11.Token.getOpSession(Token.java:241)

at sun.security.pkcs11.P11SecureRandom.engineNextBytes(P11SecureRandom.java:92)

at java.security.SecureRandom.nextBytes(SecureRandom.java:413)

at com.sun.net.ssl.internal.ssl.RandomCookie.<init>(RandomCookie.java:34)

at com.sun.net.ssl.internal.ssl.HandshakeMessage$ClientHello.<init>(Handshak eMessage.java:189)

at com.sun.net.ssl.internal.ssl.ClientHandshaker.getKickstartMessage(ClientHandsha ker.java:711)

at com.sun.net.ssl.internal.ssl.Handshaker.kickstart(Handshaker.java:522)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.jav a:1101)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImp l.java:1024)

at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:10 38)

... 45 more

ohionorthern at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 3

It looks like you are running into the same issue as this thread:

http://forum.sun.com/thread.jspa?threadID=25740&messageID=92719

I Sun is aware of, and actively investigating this issue, as the stack trace makes fairly clear, there is an issue with SSL creations between the Proxy and the Sun site. I will update the original thread

(http://forum.sun.com/thread.jspa?threadID=25740&messageID=92719)

when I know more.

-ethan

ejrider at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 4
I meant to say SSL session creation above, but the gist of what I said still applies.-ethan
ejrider at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 5
This is fine and dandy that Sun is investigating, but what do we do in the meantime when we can't install patches? These machines are production machines, not test boxes.
ohionorthern at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 6
in the meantime you can try again, as this is an intermittant issue.
ejrider at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 7

I wouldn't classify this as an intermittant issue on our end. Everything worked fine on the Solaris 10 machines until the Sun Update Connection 1.0 packages were installed as a patch using the smpatch command. Once this package was installed, none of the machines have been able to connect and update.

ohionorthern at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 8
The engineering team are currently working on resolving this issue.We hope to post an update soon.Scott
ForumModerator at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 9

Ethan and I are seriously looking into the problem.

We have (with the help of our lab administrators) reproduced a close to production environment in our lab and have been able to reproduce the problem. This helped rule out a few root causes.

We are also looking at the proxy code and have identified a couple root cause candidates. Yes, we are doing this on a Friday night. What can I say, we are engineers, and this particular problem has been stumping us for too long.

I'll post again once we have a better idea what the actual root cause is.

Frederic

LostInColorado at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 10

I had the same issue with my proxy after going to version 1.0 and it actually was the same as an old bug. Make sure you have v1.0 of both proxy software and updatemanager/connection (or whatever the name is this month). I updated my proxy sw to v1.0 after I patching to v1.0 updatemananger via the patch, not that it should matter but who knows.

In my case I write all the proxy info to a non-default filesystem/dir and the proxy wrote stuff to /var/sadm/spool when I patched it. As in the past this causes the proxy to not work and give really weird errors.

I stopped patchsvr, deleted everything in /var/sadm/spool except the cache dir (so the registration stuff doesn't get trashed) and re-started patchsvr.

I made sure all my Sol10 clients had v1.0 on them and it has worked fine ever since.

On another somewhat unrelated note, I also noticed that after uninstalling the old sol9 patchmanager and adding the new version (they are patches now), I couldn't do an smpatch without first bouncing the patchsvr. Again, when in doubt bounce patchsvr.

In general I bounce the patchsvr every six hours via cron to get around the issue where it quits responding consistently. Sure it's a hack, but until the code is solid...

I also update my proxy via wget daily to keep all the patches current. Someone said there was a way to configure v1.0 proxy to do this, but I haven't found it in the docs and it's dumb to use the clients to populate it. In case you are wondering why I do this, because it avoids the issue where the client can't get the patch via the patch proxy when it isn't cached on the proxy. In the real world we need to have the patches available when we need them.

I am using a Sol10 box with patch proxy to provide patches to Sol 8,9,10 clients without issue. The Sol10 boxes all have v1.0 on them and they are not registered since they hit the proxy patchsvr. The Sol 8/9 boxes now point to the Sol10 proxy too now that they are supported in v1.0

I use smpatch (analyze, dl, add) for all three OS's if they are working servers since we need to keep dev and prod consistent, and use smpatch update or updatemanager for the Sol10 desktops (since it's cute and gui some users like it).

All this helps keep the sysadmins I support from giving me a hard time since it actually works when they go to do their patch work. Those pro AIX and Linux admins love to trash Solaris.

Hope this helps

jb

jwbledsoe at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 11

More like a common intermittant issue - ie it's mostly broken.

I've been trying to get UpdateConnect 1.0 working with Sun Update Connection Proxy, System Edition 1.0 (119788-02) and it breaks with the following error

(CRITICAL) => com.sun.patchpro.server.ServerPatchServiceProvider@94257f <=javax.net.ssl.SSLException: java.security.ProviderException: Could not obtain session

Seems like a timeout connecting to getupdates.sun.com

Perhaps a peek at wget source or http://www.sun.com/download/sdm/download.xml would help.

SuperAlberto at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 12
Yes, we know that there is a problem. We are still trying to determine what the exact root cause is.Frederic
LostInColorado at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 13
Any progress? Its frustrating as hell trying to use the proxy stuff at the moment...
akiernan2 at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 14

We identified the root cause for this problem and we now have a work around.

The bug is described in CR #6223863 and related CRs. Basically, if the proxy used the kernel crypto modules, it could only connect up to 16 times. The fixes have been integrated into Nevada and into Solaris 10 Update 1. I don't know if a patch is going to be released against it.

The work around is simple: disable the kernel's crypto services and restart the proxy. This will force Java to use it's own crypto mechanisms rather than relying on the kernel's services.

First, determine whether the crypto services are running:

# svcs -a svc:/system/cryptosvc

STATE STIMEFMRI

online 8:21:32 svc:/system/cryptosvc:default

Stop the proxy if it is already running:

# patchsvr stop

Next, you need to stop or disable the service:

# svcadm disable svc:/system/cryptosvc

For good measure, look if the kcfd process is still running:

# ps -ef | grep kcfd

Kill it if it is still there.

Start the proxy:

# patchsvr start

From that point forward, the proxy will rely on Java's crypto code rather than the kernel's modules, and you should be able to connect back to Sun.

You can use the same work around if you are having SSL connection problems with smpacth. This could happen when you are downloading a large number of patches at the same time.

Frederic Jean

Engineering Lead, Delivery Server

LostInColorado at 2007-7-6 13:28:01 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 15

Darren Moffat (http://blogs.sun.com/darren), who has a much better understanding of how cryptography works in Solaris, pointed out that the approach suggested was quite drastic. He provided a much better workaround.

First, stop the proxy:

patchsvr stop

See if the crypto services are running:

svcs svc:/system/cryptosvc

You'll need to restart them if they are disabled:

svcadm enable svc:/system/cryptosvc

Now, you can disable the problematic component:

cryptoadm disable metaslot

And restart the proxy:

patchsvr start

I retested the proxy server in this configuration, and it is working.

My apologies for the previous drastic workaround.

Frederic

LostInColoradoa at 2007-7-21 14:57:34 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 16

What is metaslot ?

Doesnt exist on my Solaris 10 x86 server.

(udas1) 12# cryptoadm disable metaslot

Usage:

cryptoadm list [-mpv] [provider=<provider-name>] [mechanism=<mechanism-list>]

cryptoadm disable provider=<provider-name> mechanism=<mechanism-list> | random | all

cryptoadm enable provider=<provider-name> mechanism=<mechanism-list> | random | all

cryptoadm install provider=<provider-name>

cryptoadm install provider=<provider-name> [ mechanism=<mechanism-list> ]

cryptoadm uninstall provider=<provider-name>

cryptoadm unload provider=<provider-name>

cryptoadm refresh

cryptoadm start

cryptoadm stop

cryptoadm --help

Erik@Sendiaa at 2007-7-21 14:57:34 > top of Java-index,Administration Tools,Sun Update Connection-System...