All deferred patches fail

This happened today on both my Solaris 10 test servers (SPARC and x86) when I ran

`smpatch update'. The immediate patch installation went normally, but when I used

`init 6' to reboot, the console said:

Installing update 119685-10 Failed

Installing update 118371-08 Failed

Installing update 125016-02 Failed

Installing update 122032-04 Failed

Installing update 124254-03 Failed

Installing update 124250-03 Failed

Installing update 118925-05 Failed

Installing update 120469-06 Failed

Installing update 125427-01 Failed

After the reboot, sunucLog/job_history.log said:

Mar 04 13:54:22 celano Deferred Installation Log

Mar 04 13:54:22 celano 119685-10 [notice] Status Install Begin 119685-10

Mar 04 13:54:22 celano 119685-10 [ALERT] Validating patches...

Mar 04 13:54:22 celano 119685-10 [ALERT] Loading patches installed on the system...

Mar 04 13:54:22 celano 119685-10 [ALERT] Loading patches requested to install.

Mar 04 13:54:22 celano 119685-10 [ALERT] Cannot locate /119685-10.jar.dir/119685-10 to install.

Mar 04 13:54:22 celano 119685-10 [ALERT] Status Install End 119685-10 Install Update installation failed

...

for every patch. The files are present in /var/sadm/spool...

$ ll -td 119685-10.* 125427-01.*

-rw-r--r--1 rootroot83890 Mar 3 05:12 125427-01.jar

-rw-r--r--1 rootroot185604 Feb 24 05:30 119685-10.jar

Why can't it find them. This has been working up until this week.

[1544 byte] By [Gary_Millsa] at [2007-11-26 20:24:15]
# 1

Please can you run the "suc.sh" scrip on this system which can be downloaded from the link below:

https://supportfiles.sun.com/download?directory=downloads&file=suc%2esh

And paste the contents of the "suc-out" file generated.

Please can you also run the following commands:

# ls -laR /var/sadm/spool/

If the patch "119685-10" is available please can you attempt to apply it via "smpatch":

# smpatch update -i 119685-1

If this also fail please can you attempt to apply it via "patchadd":

# patchadd /var/sadm/spool/119685-1.jar

# cat /var/sadm/patch/119685-10/log

And return the output generated.

ForumModeratora at 2007-7-10 0:50:20 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 2

# env USER=root LOGNAME=root ./suc.sh

$USER: root

$LOGNAME: root

Monday, March 5, 2007 7:11:23 PM CST

celano

smpatch get:

patchpro.backout.directory-""

patchpro.baseline.directory-/var/sadm/spool

patchpro.download.directory-/var/sadm/spool

patchpro.install.types -rebootafter:reconfigafter:standard

patchpro.patch.sourcehttp://patches.cc.umanitoba.ca:3816/solaris/https://getupdates1.sun.com/

patchpro.patchset-current

patchpro.proxy.host -""

patchpro.proxy.passwd********

patchpro.proxy.port -8080

patchpro.proxy.user -""

smpatch analyze:

Error: Unable to download document : "xml/motd.xml"

Cannot connect to retrieve xml/motd.xml: Bad Request

Unable to display any message of the day notices from Sun Microsystems. Refer to the log file for additional information.

119685-10 SunOS 5.10: svc.startd patch

118371-08 SunOS 5.10: elfsign Patch

125016-02 SunOS 5.10: audit and init patch

118557-08 SunOS 5.10: platform/sun4u/kernel/drv/sparcv9/su patch

122032-04 SunOS 5.10: Update timezones patch

123252-01 SunOS 5.10: platform/SUNW,Netra-T2000 patch

124254-03 SunOS 5.10: sockfs patch

124250-03 SunOS 5.10: rpcmod patch

118925-05 SunOS 5.10: unistd header file patch

120469-06 SunOS 5.10: kerberos patch

123590-03 SunOS 5.10: PostgresSQL patch

125427-01 SunOS 5.10: ohci patch

showrev -p

119788-07

120335-04

121081-06

121118-09

121118-10

121118-11

121453-02

122231-01

123005-05

124463-02

124614-01

java -version:

java version "1.5.0_11"

Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)

Java HotSpot(TM) Client VM (build 1.5.0_11-b03, mixed mode, sharing)

/etc/release:

Solaris 10 11/06 s10s_u3wos_10 SPARC

Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.

Use is subject to license terms.

Assembled 14 November 2006

/var/sadm/system/admin/CLUSTER:

CLUSTER=SUNWCall

/usr/lib/cc-ccr/bin/ccr -g cns.assetid:

patchsvr setup -l

Patch source URL: https://getupdates1.sun.com/

Cache location: /var/sadm/spool/patchsvr

Please attach /tmp/suc-out.Z in your reply to the Sun Update Connection Technical Support Team.

# ls -laR /var/sadm/spool/

/var/sadm/spool/:

total 6162

drwxr-xr-x7 rootsys 2048 Mar 4 13:54 .

drwxr-xr-x 13 rootsys 512 Mar 4 11:31 ..

-rw-r--r--1 rootroot1119723 Feb 24 05:30 118371-08.jar

-rw-r--r--1 rootroot67694 Feb 24 05:30 118925-05.jar

-rw-r--r--1 rootroot185604 Feb 24 05:30 119685-10.jar

-rw-r--r--1 rootroot797382 Mar 3 05:12 120469-06.jar

-rw-r--r--1 rootroot334464 Feb 24 05:30 122032-04.jar

-rw-r--r--1 rootroot25061 Dec 16 09:01 123252-01.jar

drwxr-xr-x4 rootroot 512 Dec 16 09:31 123252-01.jar.dir

-rw-r--r--1 rootroot205784 Feb 24 05:30 124250-03.jar

-rw-r--r--1 rootroot164409 Mar 3 05:12 124254-03.jar

-rw-r--r--1 rootroot66323 Mar 3 05:12 125016-02.jar

-rw-r--r--1 rootroot83890 Mar 3 05:12 125427-01.jar

drwxr-xr-x6 rootsys 512 Mar 5 19:11 cache

-rw-r--r--1 rootroot1198 Feb 17 10:08 disallowed_patch_list_report

-rw-r--r--1 rootroot 440 Mar 4 09:49 jar_list

-rw-r--r--1 rootroot0 Mar 4 11:31 patch.report

drwxr-xr-x2 rootroot1536 Mar 4 11:31 patchproSequester

drwxr-xr-x2 rootsys 512 Dec 16 09:57 patchsvr

-rw-r--r--1 rootroot2015 Mar 4 09:47 smanalyze.log

-rw-r--r--1 rootroot0 Mar 4 09:45 smanalyze.log.err

-rw-r--r--1 rootroot4004 Mar 4 09:49 smdownload.log

-rw-r--r--1 rootroot7340 Mar 4 11:31 smupdate.log

drwxr-xr-x2 rootroot 512 Mar 4 13:54 sunucLog

/var/sadm/spool/123252-01.jar.dir:

total 10

drwxr-xr-x4 rootroot 512 Dec 16 09:31 .

drwxr-xr-x7 rootsys 2048 Mar 4 13:54 ..

drwxr-xr-x3 rootroot 512 Mar 4 10:02 123252-01

drwxr-xr-x2 rootroot 512 Mar 4 10:02 META-INF

/var/sadm/spool/123252-01.jar.dir/123252-01:

total 42

drwxr-xr-x3 rootroot 512 Mar 4 10:02 .

drwxr-xr-x4 rootroot 512 Dec 16 09:31 ..

-rwxr-xr-x1 rootroot 76 Jun 23 2006 .diPatch

-r--r--r--1 rootroot13272 Sep 27 2005 LEGAL_LICENSE.TXT

-rwxr-xr-x1 rootroot 183 Jun 23 2006 patchinfo

-rw-r--r--1 rootroot2668 Jul 24 2006 README.123252-01

drwxr-xr-x3 rootroot 512 Mar 4 10:02 SUNWcart200.v

/var/sadm/spool/123252-01.jar.dir/123252-01/SUNWcart200.v:

total 10

drwxr-xr-x3 rootroot 512 Mar 4 10:02 .

drwxr-xr-x3 rootroot 512 Mar 4 10:02 ..

drwxr-xr-x2 rootroot 512 Mar 4 10:02 install

-rwxr-xr-x1 rootroot 595 Jun 23 2006 pkginfo

-rwxr-xr-x1 rootroot 390 Jun 23 2006 pkgmap

/var/sadm/spool/123252-01.jar.dir/123252-01/SUNWcart200.v/install:

total 64

drwxr-xr-x2 rootroot 512 Mar 4 10:02 .

drwxr-xr-x3 rootroot 512 Mar 4 10:02 ..

-rwxr-xr-x1 rootroot4985 Jun 23 2006 checkinstall

-rwxr-xr-x1 rootroot 93 Jun 23 2006 copyright

-rwxr-xr-x1 rootroot5768 Jun 23 2006 i.none

-rwxr-xr-x1 rootroot1555 Jun 23 2006 patch_checkinstall

-rwxr-xr-x1 rootroot 824 Jun 23 2006 patch_postinstall

-rwxr-xr-x1 rootroot6703 Jun 23 2006 postinstall

-rwxr-xr-x1 rootroot7437 Jun 23 2006 preinstall

/var/sadm/spool/123252-01.jar.dir/META-INF:

total 24

drwxr-xr-x2 rootroot 512 Mar 4 10:02 .

drwxr-xr-x4 rootroot 512 Dec 16 09:31 ..

-rw-r--r--1 rootroot3820 Jul 24 2006 es-signature.rsa

-rw-r--r--1 rootroot2178 Jul 24 2006 es-signature.sf

-rw-r--r--1 rootroot2070 Jul 24 2006 manifest.mf

/var/sadm/spool/cache:

total 7166

drwxr-xr-x6 rootsys 512 Mar 5 19:11 .

drwxr-xr-x7 rootsys 2048 Mar 4 13:54 ..

drwxr-xr-x2 rootroot 512 Mar 5 19:11 Database

drwxr-xr-x2 rootroot 512 Mar 5 19:13 entitlement

-rw-r--r--1 rootroot3646470 Mar 5 19:11 http%3A%2F%2Fpatches.cc.umanitoba.ca%3A3816%2Fsolaris%2F%2Fdetectors.jar

drwxr-xr-x3 rootsys 512 Dec 13 17:49 updatemanager

drwxr-xr-x2 rootroot 512 Mar 5 19:13 xml

/var/sadm/spool/cache/Database:

total 612

drwxr-xr-x2 rootroot 512 Mar 5 19:11 .

drwxr-xr-x6 rootsys 512 Mar 5 19:11 ..

-rw-r--r--1 rootroot299517 Mar 5 19:11 http%3A%2F%2Fpatches.cc.umanitoba.ca%3A3816%2Fsolaris%2F%2FDatabase%2Fcurrent.z ip

/var/sadm/spool/cache/entitlement:

total 6

drwxr-xr-x2 rootroot 512 Mar 5 19:13 .

drwxr-xr-x6 rootsys 512 Mar 5 19:11 ..

-rw-r--r--1 rootroot 197 Mar 5 19:13 http%3A%2F%2Fpatches.cc.umanitoba.ca%3A3816%2Fsolaris%2F%2Fentitlement_client

/var/sadm/spool/cache/updatemanager:

total 6

drwxr-xr-x3 rootsys 512 Dec 13 17:49 .

drwxr-xr-x6 rootsys 512 Mar 5 19:11 ..

drwxr-xr-x2 rootsys 512 Dec 13 17:49 analysis.results

/var/sadm/spool/cache/updatemanager/analysis.results:

total 4

drwxr-xr-x2 rootsys 512 Dec 13 17:49 .

drwxr-xr-x3 rootsys 512 Dec 13 17:49 ..

/var/sadm/spool/cache/xml:

total 4

drwxr-xr-x2 rootroot 512 Mar 5 19:13 .

drwxr-xr-x6 rootsys 512 Mar 5 19:11 ..

/var/sadm/spool/patchproSequester:

total 70

drwxr-xr-x2 rootroot1536 Mar 4 11:31 .

drwxr-xr-x7 rootsys 2048 Mar 4 13:54 ..

-rw-r--r--1 rootroot 592 Dec 16 10:03 patchpro_dnld_2006.12.16@09:03:58:CST.txt

-rw-r--r--1 rootroot 592 Dec 16 10:03 patchpro_dnld_2006.12.16@09:19:55:CST.txt

-rw-r--r--1 rootroot 360 Jan 20 17:37 patchpro_dnld_2006.12.23@05:29:02:CST.txt

-rw-r--r--1 rootroot 443 Jan 20 17:37 patchpro_dnld_2006.12.30@05:02:03:CST.txt

-rw-r--r--1 rootroot 529 Jan 20 17:37 patchpro_dnld_2007.01.06@05:22:42:CST.txt

-rw-r--r--1 rootroot 543 Jan 20 17:37 patchpro_dnld_2007.01.13@04:55:57:CST.txt

-rw-r--r--1 rootroot 708 Jan 20 17:37 patchpro_dnld_2007.01.20@04:47:49:CST.txt

-rw-r--r--1 rootroot 708 Jan 20 17:37 patchpro_dnld_2007.01.20@16:53:12:CST.txt

-rw-r--r--1 rootroot 278 Feb 4 09:48 patchpro_dnld_2007.01.27@05:22:57:CST.txt

-rw-r--r--1 rootroot1148 Feb 4 09:48 patchpro_dnld_2007.02.02@18:56:48:CST.txt

-rw-r--r--1 rootroot 723 Feb 4 09:48 patchpro_dnld_2007.02.02@21:26:14:CST.txt

-rw-r--r--1 rootroot 723 Feb 4 09:48 patchpro_dnld_2007.02.03@04:58:47:CST.txt

-rw-r--r--1 rootroot 709 Feb 4 09:48 patchpro_dnld_2007.02.04@09:29:12:CST.txt

-rw-r--r--1 rootroot 581 Feb 4 14:01 patchpro_dnld_2007.02.04@10:34:48:CST.txt

-rw-r--r--1 rootroot 548 Feb 4 14:01 patchpro_dnld_2007.02.04@13:39:31:CST.txt

-rw-r--r--1 rootroot 548 Feb 4 14:01 patchpro_dnld_2007.02.04@13:57:00:CST.txt

-rw-r--r--1 rootroot 252 Feb 11 09:54 patchpro_dnld_2007.02.04@14:14:10:CST.txt

-rw-r--r--1 rootroot 252 Feb 11 09:54 patchpro_dnld_2007.02.06@10:14:00:CST.txt

-rw-r--r--1 rootroot 486 Feb 11 09:54 patchpro_dnld_2007.02.10@04:51:43:CST.txt

-rw-r--r--1 rootroot 739 Feb 11 09:54 patchpro_dnld_2007.02.11@09:37:32:CST.txt

-rw-r--r--1 rootroot 923 Feb 17 10:04 patchpro_dnld_2007.02.14@08:35:02:CST.txt

-rw-r--r--1 rootroot 608 Feb 17 10:04 patchpro_dnld_2007.02.17@05:39:31:CST.txt

-rw-r--r--1 rootroot 608 Feb 17 10:04 patchpro_dnld_2007.02.17@08:40:38:CST.txt

-rw-r--r--1 rootroot 971 Mar 4 11:31 patchpro_dnld_2007.02.24@05:32:21:CST.txt

-rw-r--r--1 rootroot1191 Mar 4 11:31 patchpro_dnld_2007.03.03@05:13:01:CST.txt

-rw-r--r--1 rootroot1191 Mar 4 11:31 patchpro_dnld_2007.03.04@09:49:37:CST.txt

-rw-r--r--1 rootroot1191 Mar 4 11:31 patchpro_dnld_2007.03.04@09:52:54:CST.txt

/var/sadm/spool/patchsvr:

total 6

drwxr-xr-x2 rootsys 512 Dec 16 09:57 .

drwxr-xr-x7 rootsys 2048 Mar 4 13:54 ..

/var/sadm/spool/sunucLog:

total 16

drwxr-xr-x2 rootroot 512 Mar 4 13:54 .P

drwxr-xr-x7 rootsys 2048 Mar 4 13:54 ..

-rw-r--r--1 rootroot4941 Mar 4 13:54 job_history.log

Patch 119685-10 is the svc.startd patch, which wants to be applied in single-user

mode. I'll try that and post another reply.

Gary_Millsa at 2007-7-10 0:50:20 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 3

I misunderstood your instructions. I see that you wanted me to to run `smpatch update '

in multi-user mode. Here's how that went:

# cd /var/sadm/spool

# smpatch update -i 119685-10

Error: Unable to download document : "xml/motd.xml"

Cannot connect to retrieve xml/motd.xml: Bad Request

Unable to display any message of the day notices from Sun Microsystems. Refer to the log file for additional information.

Update 119685-10 will not be downloaded since it already exists in the download directory.

Installing patches from /var/sadm/spool...

NOTICE: Update 119685-10 cannot be applied at this time since it is typed as "reboot immediate" which is disallowed by installation policy.

NOTICE: Patch 119685-10 cannot be installed until the next system shutdown.

/var/sadm/spool/patchpro_dnld_2007.03.05@20:19:45:CST.txt has been moved to /var/sadm/spool/patchproSequester/patchpro_dnld_2007.03.05@20:19:45:CST.txt

ID's of the updates that are disallowed by installation policy have been

written to file

/var/sadm/spool/disallowed_patch_list

One or more updates that you installed requires a system shutdown to activate it. To initiate the system shutdown, you must use one of the following commands:

o Drop to the firmware prompt - init 0 or shutdown -i 0

o Power down the system - init 5 or shutdown -i 5

o Restart the system - init 6 or shutdown -i 6

# init 0

Her's the relevant part of the console log:

Mar 5 20:19:47 celano root: => Thread[Thread-20,5,main] <=Problem installing patches: NOTICE: Patch 119685-10 cannot be installed until the next system shutdown.

Mar 5 20:19:47 celano root: => Thread[Thread-20,5,main] <=Problem installing patches: NOTICE: Patch 119685-10 cannot be installed until the next system shutdown.

svc.startd: The system is coming down. Please wait.

svc.startd: 106 system services are now being stopped.

Mar 5 20:24:46 celano syslogd: going down on signal 15

Installing update 119685-10 Succeeded

umount: /opt/csw busy

umount: /opt busy

svc.startd: The system is down.

syncing file systems... done

Program terminated

{1} ok boot

This time it worked! What could be different?

Gary_Millsa at 2007-7-10 0:50:20 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 4

I tried `smpatch update' for the rest of the deferred patches. All of them applied

normally:

Installing update 118371-08 Succeeded

Installing update 125016-02 Succeeded

Installing update 118557-08 Succeeded

Installing update 122032-04 Succeeded

Installing update 124254-03 Succeeded

Installing update 124250-03 Succeeded

Installing update 118925-05 Succeeded

Installing update 120469-06 Succeeded

Installing update 125427-01 Succeeded

I did notice that the format of the disallowed_patch_list file has changed:

118371-08:/var/sadm/spool

125016-02:/var/sadm/spool

118557-08:/var/sadm/spool

122032-04:/var/sadm/spool

124254-03:/var/sadm/spool

124250-03:/var/sadm/spool

118925-05:/var/sadm/spool

120469-06:/var/sadm/spool

125427-01:/var/sadm/spool

I wonder if that's the reason why the patches all failed the last time around?

Gary_Millsa at 2007-7-10 0:50:20 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 5

It's hard to say what the problem was - since 119685-10 patches part of svc, I'd be inclined to think there was an issue on shutdown which stopped the patch process working. That said, I can't find any support cases where 119685-10 has caused any problems.

Are any other machines seeing this problem?

Message was edited by:

ForumModerator

ForumModeratora at 2007-7-10 0:50:20 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 6

Patch 119685-10 didn't cause this problem. Any deferred patch would have the same

problem. It's caused by patch 121119-11 (in the x86 case). My Solaris 10 x86 test server

behaved exactly the same as my Solaris 10 SPARC server: when I ran `smpatch update',

all of the immediate patches succeeded but all of the deferred patches failed.

That's because patch 121119-11 was installed as one of the immediate patches.

It updates the /lib/svc/method/installupdates to use the new format of the disallowed_patch_list file.

However, that smpatch run has already generated a disallowed_patch_list file in the

old format. The new method then assumes that all patches are located in the root

directory, instead of /var/sadm/spool. You will need to withdraw this patch and create

one with a method file that supports both formats.

The workaround is to install 121119-11 (or its SPARC equivalent) with `smpatch add -i'

before installing the remainder of the patches with `smpatch update'.

Gary_Millsa at 2007-7-10 0:50:20 > top of Java-index,Administration Tools,Sun Update Connection-System...
# 7
I saw the same problem on my machine. But of course re-running smpatch update and rebooting fixed everything.
davidholmesa at 2007-7-10 0:50:20 > top of Java-index,Administration Tools,Sun Update Connection-System...