Local patch server confused again
Starting earlier this week, Solaris 10 clients that connect to our local patch server
only report that patch 121118-08 is required. Running `smpatch analyze' a week
ago showed many more patches. After 121118-08 is applied, another `smpatch
analyze' says `No patches required.'. One of these clients needed 139 patches before,
and now needs none! What happened?
Solaris 10 machines that don't go through the local patch server are behaving
normally. Restarting the local patch server didn't help. The local patch server's
URL is `https://getupdates1.sun.com/solaris/'. It hasn't been changed for some time,
and was working a week ago.
# 1
There was an issue with the current.zip file earlier in the week which caused this behaviour. The file has been corrected on our side so it's likely that your hosts are using a cached version for some reason.
Clear out the /var/sadm/spool/patchsvr/Database directory on the SunUC proxy and the /var/sadm/spool/cache/Database on the SunUC proxy and the clients to force a re-download
# 2
That doesn't seem to help. Both the server and client download a new copy ofthe file that's only 1318 bytes in length. Where's it coming from?
# 3
I noticed that the patchsvr patch reset patchsvr cache location!but resetting it didn't help, I still says no patches required. What's the deal here, anyone heard of QA at Sun?Message was edited by: jwbledsoe
# 4
I even tried resetting patchsvr, seems it won't even do that.# patchsvr setup -dProperty not defined: cns.patchsrv.cachelocation#Anyone got any idea what the fix is?
# 5
You can clear the directory
# rm -rf /var/sadm/spool/cache/*
then check the source url on proxy :
# patchsvr setup -l
or on clients if no proxy is used :
# smpatch get
this should conform to :
The Update Connection Proxy should contain the patchsource URL :
https://getupdates1.sun.com/
provided all client systems of this Update Connection Proxy are Solaris 10.
If they are of a mixed Solaris version, please set it to
https://getupdates.sun.com/solaris/
The above applies to the Update Connection Proxy _only_.
The output of smpatch get on the LPS may be : https://getupdates1.sun.com/
Notice the trailing slash ("/") in all cases.
Also note the clearing of /var/sadm/spool/cache above should NOT remove the cache dir itself , it should also be done on all the failing clients as well.
Let us know how you get on.
Mod.
# 6
if I clear /var/sadm/spool/cache on my Solaris 10 box which is the LPS, then I'll loose my registration information. At least that was how it was in the past.
# 7
on the LPS server I cleared the cache dir
did an analyze ok, download ok
verified patchsvr setup was ok for Soalris 10 clients and cleared the patchsvr cache dir. restarted patchsvr
and tried a solaris 10 client which still said no patches required
in the patchsvr cache dir I see this:
ljcqs152# patchsvr setup -l
Patch source URL: https://getupdates1.sun.com/
Cache location: /jumpstart/patches/patchsvr
Web proxy host name: usgproxy.cnf.com
Web proxy port number: 8888
ljcqs152#
ljcqs152# ls -l /jumpstart/patches/patchsvr
total 7654
-rw-r--r--1 rootroot3630182 Dec 15 18:39 %2Fsolaris%2F%3Faction%3DdownloadRealizationDetectors%26name%3Ddetectors%26vers ion%3D2.1
-rw-r--r--1 rootroot 190 Dec 15 18:38 %2Fsolaris%2F%3Faction%3DgetEntitlement%26name%3D%26version%3D2.1
-rw-r--r--1 rootroot1318 Dec 15 18:38 %2Fsolaris%2F%3Faction%3DgetFile%26name%3DDatabase%2Fcurrent.zip%26version%3D2. 1
drwxr-xr-x2 rootroot257024 Dec 15 04:27 Patches
ljcqs152#
the Patches dir is where the .jar files went (previously, when it worked)
# 8
Hi again.
Odd. I would expect to see a directory listing like:
$ ls -l /var/sadm/spool/patchsvr
total 70
drwxr-xr-x2 rootroot13312 Dec 16 02:50 Database
drwxr-xr-x4 rootroot2048 Dec 15 22:45 Misc
drwxr-xr-x2 rootroot6144 Oct 31 02:11 Patches
drwxr-xr-x2 rootroot 512 Nov 7 19:31 ReadMe
drwxr-xr-x2 rootroot 512 Dec 13 21:30 category
drwxr-xr-x2 rootroot 512 Dec 13 21:29 collection
drwxr-xr-x2 rootroot11264 Dec 16 02:51 entitlement
in your patch server's cache.
Could you unzip the %2Fsolaris%2F%3Faction%3DgetFile%26name%3DDatabase%2Fcurrent.zip%26version%3D2. file, and post its contents (assuming it's small enough to post)?
Mod.
# 9
Hmm, I just changed the URL on the patc server from:
https://getupdates.sun.com/solaris/
to:
https://getupdates1.sun.com/
and now it's working again. I didn't remove any files on the server or on the clients.
The server did download a much larger current.zip file:
# ll /var/sadm/spool/patchsvr/Database/*
-rw-r--r--1 rootroot1318 Dec 15 20:30 /var/sadm/spool/patchsvr/Database/https%3A%2F%2Fgetupdates.sun.com%2Fsolaris%2F %2FDatabase%2Fcurrent.zip
-rw-r--r--1 rootroot292708 Dec 16 08:53 /var/sadm/spool/patchsvr/Database/https%3A%2F%2Fgetupdates1.sun.com%2F%2FDataba se%2FDatabase%2Fcurrent.zip
Here's the contents of the bad one:
# unzip -l /var/sadm/spool/patchsvr/Database/*
Archive: /var/sadm/spool/patchsvr/Database/https%3A%2F%2Fgetupdates.sun.com%2Fsolaris%2F %2FDatabase%2Fcurrent.zip
LengthDateTimeName
---
91 12-12-06 14:13patchlist.properties
4514 12-12-06 14:13patchlist.delimited
-
46052 files
# 10
It's good to hear that it's working for you.I note the dates from the bad current.zip are 12-12-06 - is it possible that your web proxy cached the bad current.zip from that date?Mod.
# 11
I deleted that file several times from the patch server's database. It kept downloadinga new copy from Sun's patch server. After I changed the URL, it downloaded a newand much larger copy.
# 12
Here's one from today where I deleted everything in the LPS cache dir except the old Patches dir.
I had another admin watch our Internet proxy and LPS did go to https://getupdates1.sun.com/ to get these files.
ljcqs152# ls -l
total 534
-rw-r--r--1 rootroot 190 Dec 17 04:37 %2Fsolaris%2F%3Faction%3DgetEntitlement%26name%3D%26version%3D2.1
-rw-r--r--1 rootroot1318 Dec 17 04:37 %2Fsolaris%2F%3Faction%3DgetFile%26name%3DDatabase%2Fcurrent.zip%26version%3D2. 1
drwxr-xr-x2 rootroot257024 Dec 17 04:29 Patches
ljcqs152#
ljcqs152# unzip -l %2Fsolaris%2F%3Faction%3DgetFile%26name%3DDatabase%2Fcurrent.zip%26version%3D2. 1
Archive: %2Fsolaris%2F%3Faction%3DgetFile%26name%3DDatabase%2Fcurrent.zip%26version%3D2. 1
LengthDateTimeName
---
91 12-12-06 12:13patchlist.properties
4514 12-12-06 12:13patchlist.delimited
-
46052 files
ljcqs152#
ljcqs152# cat %2Fsolaris%2F%3Faction%3DgetEntitlement%26name%3D%26version%3D2.1
Solaris10
SolarisAllUpdates
Solaris10Security
SolarisSecurityUpdates
Solaris10Security
SolarisSecurityUpdates
SolarisDataIntegrityUpdates
SolarisHardwareUpdates
SolarisUtilityUpdates
Public
ljcqs152#
# 13
Here's a bit more data. Clearly it's the LPS that's messed up. As you can see above the LPS is getting an old current.zip but the LPS host actually gets the correct one when it goes to the same site to get the same file.
ljcqs152# pwd
/var/sadm/spool/cache/Database
ljcqs152# ls -l
total 592
-rw-r--r--1 rootroot292708 Dec 17 04:25 https%3A%2F%2Fgetupdates1.sun.com%2F%2FDatabase%2Fcurrent.zip
ljcqs152# unzip -l https%3A%2F%2Fgetupdates1.sun.com%2F%2FDatabase%2Fcurrent.zip
Archive: https%3A%2F%2Fgetupdates1.sun.com%2F%2FDatabase%2Fcurrent.zip
LengthDateTimeName
---
93 12-14-06 23:14patchlist.properties
1859331 12-14-06 23:56patchlist.delimited
-
18594242 files
ljcqs152# smpatch get
patchpro.backout.directory-""
patchpro.baseline.directory-/var/sadm/spool
patchpro.download.directory-/var/sadm/spool
patchpro.install.types -rebootafter:reconfigafter:standard
patchpro.patch.source-https://getupdates1.sun.com/
patchpro.patchset-current
patchpro.proxy.host usgproxy.cnf.com""
patchpro.proxy.passwd********
patchpro.proxy.port 88888080
patchpro.proxy.user "" ""
ljcqs152#
ljcqs152#
ljcqs152#
ljcqs152# patchsvr setup -l
Patch source URL: https://getupdates1.sun.com/
Cache location: /jumpstart/patches/patchsvr
Web proxy host name: usgproxy.cnf.com
Web proxy port number: 8888
ljcqs152#
# 14
I'm wondering why the current.zip is called '%2Fsolaris%2F%3Faction%3DgetFile%26name%3DDatabase%2Fcurrent.zip%26version%3D2 . 1'. It should be called 'https%3A%2F%2Fgetupdates1.sun.com%2F%2FDatabase%2Fcurrent.zip'.
This would seem to indicate that there's something not quite right.
Could you post the output from the following on the _client_ which was used when the LPS requested the incorrect/small DB file?
# smpatch get
and on the LPS
# smpatch get
# patchsvr setup -l
Mod.
# 15
Here's the client info, the two from the LPS server are in the preceeding post.
ljcqs171# smpatch get
patchpro.backout.directory-""
patchpro.baseline.directory-/var/sadm/spool
patchpro.download.directory-/var/sadm/spool
patchpro.install.types -rebootafter:reconfigafter:standard
patchpro.patch.sourcehttp://ljcqs152.cnf.com:3816/solaris/https://getupdates1.sun.com/
patchpro.patchset-current
patchpro.proxy.host -""
patchpro.proxy.passwd********
patchpro.proxy.port -8080
patchpro.proxy.user -""
ljcqs171# smpatch analyze
No patches required.
ljcqs171#
It should be noted that I am trying to keep it simple by only working with Solaris 10 clients and LPS, but in the past this LPS has served up for Solaris 8/9/10 without issue. As in before Sun broke LPS last week!
# 16
Everything looks fine.
Could you run the following on the LPS, so we can see the patch level:
# showrev -p | awk '/Patch: (123005|121453|121118|120335|121081|121563|122231|119788)/ {print $2}'
# showrev -p | awk '/Patch: (123006|121454|121119|120336|121082|121564|122232|119789)/ {print $2}'
(I'm the first is for SPARC, the second for x86).
Mod.
# 17
still get the same small file today. Thanks for staying on this, I need to get it fixed.
ljcqs152: showrev -p | awk '/Patch: (123005|121453|121118|120335|121081|121563|122231|119788)/ {print $2}'
120335-01
121453-02
119788-01
119788-02
121118-03
121118-05
121118-06
121118-08
119788-07
120335-02
120335-04
121081-03
121081-05
121081-02
121563-02
122231-01
ljcqs152:
# 18
something definately wrong with our LPS server. In the past I was sure that if the patchsvr directory didn't exist, that it created it and subdirs as needed. As you can see above our LPS isn't doing that either. Not sure if it's a related symptom or not, but wanted to mention it.
Here's another server I recently patched, so figured I try a little test.
ljcqs171# ls /var/sadm/spool/patchsvr
/var/sadm/spool/patchsvr: No such file or directory
ljcqs171# smpatch set patchpro.patch.source=http://localhost:3816/solaris/
This is a client so it can't connect directly to Sun, but I only want to see if it creates the dirs, which it did. (see below)
ljcqs171# smpatch analyze
Failure: Cannot connect to retrieve detectors: Internal Server Error
ljcqs171# ls -l /var/sadm/spool/patchsvr
total 4
drwxr-xr-x2 rootroot 512 Dec 18 14:37 Database
drwxr-xr-x2 rootroot 512 Dec 18 14:37 Misc
Message was edited by:
jwbledsoe
# 19
Just a bump to this thread, I am having exactly the same problem, down to filesizes as well.
Details:
[hpdepot][/var/sadm/spool]# patchsvr setup -l
Patch source URL: https://getupdates1.sun.com/
Cache location: /export/spool/patchsvr/
[hpdepot][/var/sadm/spool]# ls -l /export/spool/patchsvr/
total 446
drwxr-xr-x2 rootsys78336 Dec 13 15:13 Database.bak
drwxr-xr-x2 rootsys46080 Dec 13 14:34 Misc.bak
drwxr-xr-x2 rootsys23552 Dec 13 10:20 Patches
drwxr-xr-x2 rootroot 512 Jul 14 09:26 ReadMe
drwxr-xr-x2 rootroot 512 Aug 30 14:14 category.bak
drwxr-xr-x2 rootroot 512 Aug 30 14:14 collection.bak
drwxr-xr-x2 rootsys76800 Dec 13 15:15 entitlement.bak
[hpdepot][/var/sadm/spool]# ls -l cache/
total 0
Clean slate. Lets use smpatch with patchpro.patch.source=https://getupdates1.sun.com/
(After analyze, which returns a list of needed patches)
[hpdepot][/var/sadm/spool]# ls -l cache/ total 7124
drwxr-xr-x2 rootsys 512 Dec 18 14:52 Database
drwxr-xr-x2 rootsys 512 Dec 18 14:53 entitlement
-rw-r--r--1 rootsys3630182 Dec 18 14:52 https%3A%2F%2Fgetupdates1.sun.com%2F%2Fdetectors.jar
[hpdepot][/var/sadm/spool]# ls -l cache/Database/
total 592
-rw-r--r--1 rootsys292708 Dec 18 14:52 https%3A%2F%2Fgetupdates1.sun.com%2F%2FDatabase%2Fcurrent.zip
[hpdepot][/var/sadm/spool]# ls -l /export/spool/patchsvr/
(identical to above - __Nothing was cached by the LPS__)
Now lets change patchpro.patch.source=http://localhost:3816/solaris/
(After running smpatch analyze, which now says "No patches required.")
[hpdepot][/var/sadm/spool]# ls -l cache/Database/ total 596
-rw-r--r--1 rootsys 1318 Dec 18 14:57 http%3A%2F%2Fhpdepot.ucsd.edu%3A3816%2Fsolaris%2F%2FDatabase%2Fcurrent.zip
-rw-r--r--1 rootsys292708 Dec 18 14:52 https%3A%2F%2Fgetupdates1.sun.com%2F%2FDatabase%2Fcurrent.zip
[hpdepot][/var/sadm/spool]# ls -l /export/spool/patchsvr/ total 7572
-rw-r--r--1 rootsys3630182 Dec 18 14:57 %2Fsolaris%2F%3Faction%3DdownloadRealizationDetectors%26name%3Ddetectors%26vers ion%3D2.1
-rw-r--r--1 rootsys 190 Dec 18 14:58 %2Fsolaris%2F%3Faction%3DgetEntitlement%26name%3D%26version%3D2.1
-rw-r--r--1 rootsys 1318 Dec 18 14:57 %2Fsolaris%2F%3Faction%3DgetFile%26name%3DDatabase%2Fcurrent.zip%26version%3D2. 1
... (identical to above)
So, I'm receiving the exact same 1318 byte current.zip, and the LPS is caching everything straight in /export/spool/patchsvr instead of in the Database | Misc etc. directories as well.
Where on earth is this current.zip coming from?
# 20
keep in mind that we already verified the small file is coming from Sun, we just don't know why it's sending that old file from the 12th and not the newer one.
unzip -l the two different files, you'll see what I mean when you see the date and the small one is the offending file from before they thought they fixed the issue.
# 21
I have also the same problem:Patch source URL: https://getupdates1.sun.com/Cache location: /patchsvr/Solaris 10 hosts smpatch analyzeNo patches required...
# 22
> There was an issue with the current.zip file earlier
> in the week which caused this behaviour. The file
> has been corrected on our side so it's likely that
> your hosts are using a cached version for some
> reason.
>
> Clear out the /var/sadm/spool/patchsvr/Database
> directory on the SunUC proxy and the
> /var/sadm/spool/cache/Database on the SunUC proxy and
> the clients to force a re-download
Whouldn't be better to add something like "smpatch clear" which will clear
the cache and all related things ?
przemol
# 23
> I'm wondering why the current.zip is called
> '%2Fsolaris%2F%3Faction%3DgetFile%26name%3DDatabase%2F
> current.zip%26version%3D2. 1'. It should be called
> 'https%3A%2F%2Fgetupdates1.sun.com%2F%2FDatabase%2Fcur
> rent.zip'.
[...]
The same is happening on two machines I use, one at home, the other on my workplace
after installing the latest updates.
The subdirs {Database,Misc,Patches,...} are no longer used, instead these garbled filenames. And java throws an IO Exceptions, in the log you read filename too long.
e.g. :
Dec 12, 2006 2:28:02 PM com.sun.swup.updateserver.handler.CachingProxyHandler handleRequest
SEVERE: File name too long
java.io.IOException: File name too long
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:850)
at com.sun.swup.updateserver.handler.CachingProxyHandler.createCacheFile(CachingPr oxyHandler.java:158)
at com.sun.swup.updateserver.handler.CachingProxyHandler.getOutputStream(CachingPr oxyHandler.java:184)
at com.sun.swup.updateserver.handler.CachingProxyHandler.handleRequest(CachingProx yHandler.java:93)
at com.sun.swup.updateserver.UpdateServlet.processRequest(UpdateServlet.java:80)
at com.sun.swup.updateserver.UpdateServlet.doPost(UpdateServlet.java:115)
># smpatch get
Here are mine :
tcsh@exot-2207 [~] smpatch get
patchpro.backout.directory-""
patchpro.baseline.directory-/var/sadm/spool
patchpro.download.directory-/var/sadm/spool
patchpro.install.types rebootafter:reconfigafter:rebootimmediate:reconfigimmediate:singleuser:standard rebootafter:reconfigafter:standard
patchpro.patch.sourcehttp://corp-2200:3816/solaris/ https://getupdates1.sun.com/
patchpro.patchset-current
patchpro.proxy.host -""
patchpro.proxy.passwd********
patchpro.proxy.port -8080
patchpro.proxy.user -""
[...]
> # patchsvr setup -l
These look different since the last update
tcsh@corp-2200 [/export/corp-2200a/Patches] patchsvr setup -l
Patch source URL: https://getupdates1.sun.com/
Cache location: /export/corp-2200a/Patches/
Nothing else here.
Both machines, running Sol 10 x86 06/06.
# 24
In another thread, I was told that the patchpro.patchset setting to smpatch can be one of the following; patchdb, patchdb1 and recommended.
Is this not the case for the latest version of smpatch? I see in the above output that it is set to "current".
Has anyone tried to see whether changing that makes a difference?
Sally.
# 25
> In another thread, I was told that the
> patchpro.patchset setting to smpatch can be one of
[...]
> Has anyone tried to see whether changing that makes a
> difference?
I might be wrong here but this seems only to be a filter which defines a set of patches
being displayed/chosen.
# 26
Granted that there are problems with the patch server not operating properly, but I thought the original problem was that no patches were being listed to be applied.If it's not significant, then just ignore my message.
# 27
I think those settings are for Solaris 8/9 boxes. At least 'recommended' is, so it's not applicable to most of the posts in this thread.
# 28
Hi All
Please check that the following shows the most up to date Java SDK (not just JRE) and the correct link to it :
# java -version
# ls -ld /usr/java
Additionally, are any of your patch caches or download directories on NFS mounted filesystems or filesystems near capacity?
# df -kl < is it shown, is it almost full if it is.
Mod.
# 29
Forgot to say ...Java may be found here : http://java.sun.com/javase/downloads/index.jspYou want the "JDK 6" section.Mod.
# 30
I just updated to Java 6 - no help
[hpdepot][/src]# ls -ld /usr/java
lrwxrwxrwx1 rootsys29 Dec 19 15:43 /usr/java -> /software/common/jdk/usr/java
[hpdepot][/src]# ls -ld /software/common/jdk/usr/java
lrwxrwxrwx1 rootother 12 Dec 19 15:29 /software/common/jdk/usr/java -> jdk/jdk1.6.0
[hpdepot][/src]# java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing)
[hpdepot][/src]# smpatch analyze
No patches required.
[hpdepot][/src]# ls -l /var/sadm/spool/cache/Database/
total 4
-rw-r--r--1 rootsys 1400 Dec 19 15:45 http%3A%2F%2Fhpdepot.ucsd.edu%3A3816%2Fsolaris%2F%2FDatabase%2Fcurrent.zip
The file size increased slightly...
No problem with filesystems either.
/export is at 72%, /var is at 48%
# 31
Check the cache on the SunUC proxy. The listing you provided (from a client) indicates that the file provided by the proxy is the corrupt one.
Assuming your SunUC proxy cache location is the default the current.zip file will be located in /var/sadm/spool/patchsvr/Database
If they too are small remove them all and run an analyze from a client host again.
# 32
Hi,
We've been seeing this "No patches required" problem for the past couple of weeks, the LPS itself was reporting the expected list.
However when "smpatch analyze" was run at 3.00 GMT today all the clients returned the expected lists again.
I know I didn't change anything to fix it because I was out of the office yesterday.
Steve
# 33
[...]
> You want the "JDK 6" section.
I just installed it on the lps system , now something new happens :
tcsh@corp-2200 [/export/corp-2200a/Patches] ll *evok*
-rw-r--r--1 rootroot0 Dec 20 16:08 %2Fsolaris%2F%3Faction%3DisCertificateRevoked%26serialNumber%3D335544443
-rw-r--r--1 rootroot0 Dec 20 16:08 %2Fsolaris%2F%3Faction%3DisCertificateRevoked%26serialNumber%3D8349693264234139 878719408387361815
coincident ? Or do I miss something.
BTW:
The lps is running sol 10 x86, the client ist running sol9 sparc.
tcsh@corp-2200 [/export/corp-2200a/Patches] patchsvr setup -l
Patch source URL: https://getupdates1.sun.com/
changing the patch source url to the one indicated in another post ( referring to a mixed setup of clients )
https://getupdates.sun.com/solaris/
leaded to this on the client :
smpatch update0
com.sun.patchpro.model.PatchProRuntimeException: Unexpected throwable
at com.sun.patchpro.cli.PatchServices.waitForThread(PatchServices.java:1284)
at com.sun.patchpro.cli.PatchServices.installPatches(PatchServices.java:1121)
at com.sun.patchpro.cli.PatchServices.main(PatchServices.java:510)
Caused by:
java.lang.Throwable: ERROR: PatchDownloadIOException.MESSAGE
reverting the settings on the lps , the client was able to get the patches.
# 34
I had everything in the cache cleared on both the proxy and the client for my above post. Still getting corrupt cache files, and they're still coming with those garbled long names.
As a temporary fix, I got the correct cache file from https://getupdates1.sun.com/, ran analyze on a client, then replaced the garbled cache file with the correct one.
[hpdepot][/export/spool/patchsvr]# rm %2Fsolaris%2F%3Faction%3DgetFile%26name%3DDatabase%2Fcurrent.zip%26version%3D2. 1
[hpdepot][/export/spool/patchsvr]# mv https%3A%2F%2Fgetupdates1.sun.com%2F%2FDatabase%2Fcurrent.zip %2Fsolaris%2F%3Faction%3DgetFile%26name%3DDatabase%2Fcurrent.zip%26version%3D2. 1
# 35
Ok I think we've gone beyond the point at which we'll be able to troubleshoot this over the forum since we'll need to get some quite large debug logs to troubleshoot this further. As such can I recommend that if you have outstanding issues relating to this thread that you raise a case with Sun so we can continue to investigate.
When raising the case (or in the initial communication with the engineer who picks it up) please make reference to or provide a link to this thread so that they can see the troubleshooting steps already performed (be sure to let them know what your forum id is too then so they can pick out your data).
# 36
All,
IF the client has 121118-08 or 121119-08 or higher installed
AND IF the SUC proxy has 119788-07 or 119789-07
THEN the patchsource on the client should be
http://my_patch_server:3816/
instead of
http://my_patch_server:3816/solaris/
Correcting this setting on the clients should resolve the issue of the current.zip file you are getting being small.
The change was made in patch 119788-07/119789-07 and is documented in the respective readme files.
