Still SSH config mistakes evident during creation of solaris image profile

Well, I can SSH to and fro and back from my two clients from both sides, yet still I am getting a "Host key verification failed ( i have enabled SSH debug on all output) when trying to create a Solaris Image Profile:

....

e yes -A 10.0.2.2 /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/bin/.jvm -Xmx128m -Drox_home_dir=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server -classpath /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/../common/lib/cr_bs.ja r com.raplix.rolloutexpress.node.bootstrap.BootStrap -verbose com.raplix.rolloutexpress.SshProxy -Dnet.local.ip=10.0.2.2 -Dnet.local.port=80001 -Dnet.remote.ip=10.0.2.3 -Dnet.remote.port=98304 -Drox_config_dir=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/confi g/proxy/config

2006-12-22 13:28:49,078 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - Host key verification failed.

...

On the downstream machine, here is some ROX output from /var/tmp I believe was from the exection:

cat ROXAgent46971.log

looking for :/opt/SUNWn1sps/N1_Service_Provisioning_System/agent/../common/lib/solaris/libn ative.so

2006-12-22 13:28:40,080 DEBUG [main] com.raplix.rolloutexpress.net.transport.SshClientConnectionHandler (SshClientConnectionHandler.java:78) - Instantiated port allocator:com.raplix.rolloutexpress.net.transport.SequentialSshPortAllocator@10 a6ae2

2006-12-22 13:39:18,924 ERROR [Reaper] com.raplix.rolloutexpress.command.stickydata.ReaperContext (RunnableContext.java:166) - Execution ended abnormally. (025075)

java.lang.InterruptedException: sleep interrupted

at java.lang.Thread.sleep(Native Method)

at com.raplix.util.threads.RegularTaskContext.safeRun(RegularTaskContext.java:58)

at com.raplix.util.threads.RunnableContext.run(RunnableContext.java:196)

at java.lang.Thread.run(Thread.java:534)

any ideas would be welcomed.... except that I have to generate SSH keys for root :-)

Pete

[2037 byte] By [peteziua] at [2007-11-26 13:23:28]
# 1
Still trying to debug. How can one tell what user shell the command is running under? (guess this was cut off in the std. out?)Pete
peteziua at 2007-7-7 17:55:58 > top of Java-index,Administration Tools,N1 Service Provisioning System...
# 2

Below is an attempt to test some connectivity. "provision" is my physical MS host. "osprovision" is my physical host for OSP and JET. The commands are being issued from host "provision" (10.0.2.2 - MS):

bash-3.00$ ssh -A 10.0.2.3 hostname

osprovisioner

bash-3.00$ ssh -A 10.0.2.3 ssh -A 127.0.0.1 /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/cli/bin/cr_cli -cmd hdb.h.la -u admin -p ********

2006-12-22 15:03:11,138 DEBUG [main] com.raplix.rolloutexpress.net.transport.SshClientConnectionHandler (SshClientConnectionHandler.java:78) - Instantiated port allocator:com.raplix.rolloutexpress.net.transport.SequentialSshPortAllocator@d6 a05e

2006-12-22 15:03:11,813 DEBUG [main] com.raplix.rolloutexpress.net.transport.SshClientConnectionHandler (SshClientConnectionHandler.java:133) - Setting up connTo:ConnectionSetupAttribute:ipAddress10.0.2.2:port:80001:type:ssh:parameter s:appType=proxy,cprefix=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server ,appargs=-Drox_config_dir=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/serv er/config/proxy/config

2006-12-22 15:03:11,818 DEBUG [main] com.raplix.rolloutexpress.net.transport.SshClientConnectionHandler (SshClientConnectionHandler.java:159) - parm:appType=proxy

2006-12-22 15:03:11,819 DEBUG [main] com.raplix.rolloutexpress.net.transport.SshClientConnectionHandler (SshClientConnectionHandler.java:159) - parm:cprefix=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server

2006-12-22 15:03:11,821 DEBUG [main] com.raplix.rolloutexpress.net.transport.SshClientConnectionHandler (SshClientConnectionHandler.java:159) - parm:appargs=-Drox_config_dir=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2 /server/config/proxy/config

2006-12-22 15:03:11,827 DEBUG [main] com.raplix.rolloutexpress.net.transport.SshClientConnectionHandler (SshClientConnectionHandler.java:184) - AppArgs:-Drox_config_dir=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/serv er/config/proxy/config

2006-12-22 15:03:11,829 DEBUG [main] com.raplix.rolloutexpress.net.transport.SshClientConnectionHandler (SshClientConnectionHandler.java:236) - Adding sshArgs:[-o, BatchMode yes, -A]

2006-12-22 15:03:11,831 DEBUG [main] com.raplix.rolloutexpress.net.transport.SshClientConnectionHandler (SshClientConnectionHandler.java:295) - Adding jvmArgs:[-Xmx128m, -Drox_home_dir=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server]

2006-12-22 15:03:11,834 DEBUG [main] com.raplix.rolloutexpress.net.transport.SshClientConnectionHandler (SshClientConnectionHandler.java:319) - Adding appArg:[-Drox_config_dir=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/serv er/config/proxy/config]

2006-12-22 15:03:11,836 INFO [main] com.raplix.rolloutexpress.net.transport.SshClientConnectionHandler (SshClientConnectionHandler.java:260) - Executing:ssh -o BatchMode yes -A 10.0.2.2 /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/bin/.jvm -Xmx128m -Drox_home_dir=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server -classpath /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/../common/lib/cr_bs.ja r com.raplix.rolloutexpress.node.bootstrap.BootStrap -verbose com.raplix.rolloutexpress.SshProxy -Dnet.local.ip=10.0.2.2 -Dnet.local.port=80001 -Dnet.remote.ip=10.0.2.3 -Dnet.remote.port=98304 -Drox_config_dir=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/confi g/proxy/config

2006-12-22 15:03:12,607 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - Starting with Homedir='/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server' MainClass='com.raplix.rolloutexpress.SshProxy' Args='-Dnet.local.ip=10.0.2.2' '-Dnet.local.port=80001' '-Dnet.remote.ip=10.0.2.3' '-Dnet.remote.port=98304' '-Drox_config_dir=/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/conf ig/proxy/config'

2006-12-22 15:03:12,608 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - Common upgrade dir does not exist:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/upgrade

2006-12-22 15:03:12,609 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - App Upgrade dir does not exist:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/upgrade

2006-12-22 15:03:12,609 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - jars being used...

2006-12-22 15:03:12,610 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/ECS.jar

2006-12-22 15:03:12,610 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/activation.jar

2006-12-22 15:03:12,611 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/bootstrap.jar

2006-12-22 15:03:12,611 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/catalina.jar

2006-12-22 15:03:12,612 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/cr_bs.jar

2006-12-22 15:03:12,612 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/jasper-compiler.ja r

2006-12-22 15:03:12,613 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/jasper-runtime.jar

2006-12-22 15:03:12,613 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/log4j.jar

2006-12-22 15:03:12,614 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/mail.jar

2006-12-22 15:03:12,614 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/naming-common.jar

2006-12-22 15:03:12,615 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/naming-resources.j ar

2006-12-22 15:03:12,615 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/postgresql.jar

2006-12-22 15:03:12,616 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/rox.jar

2006-12-22 15:03:12,616 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/servlet.jar

2006-12-22 15:03:12,618 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/servlets-common.ja r

2006-12-22 15:03:12,618 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/servlets-default.j ar

2006-12-22 15:03:12,619 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/servlets-invoker.j ar

2006-12-22 15:03:12,619 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/sps-compSDK.jar

2006-12-22 15:03:12,621 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/struts.jar

2006-12-22 15:03:12,621 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/xerces.jar

2006-12-22 15:03:12,622 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/xml-apis.jar

2006-12-22 15:03:12,622 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - Set java.class.path to /opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/ECS.jar:/opt/SUNWn 1sps/N1_Service_Provisioning_System_5.2/common/lib/activation.jar:/opt/SUNWn1sps /N1_Service_Provisioning_System_5.2/server/lib/bootstrap.jar:/opt/SUNWn1sps/N1_S ervice_Provisioning_System_5.2/server/lib/catalina.jar:/opt/SUNWn1sps/N1_Service _Provisioning_System_5.2/common/lib/cr_bs.jar:/opt/SUNWn1sps/N1_Service_Provisio ning_System_5.2/server/lib/jasper-compiler.jar:/opt/SUNWn1sps/N1_Service_Provisi oning_System_5.2/server/lib/jasper-runtime.jar:/opt/SUNWn1sps/N1_Service_Provisi oning_System_5.2/common/lib/log4j.jar:/opt/SUNWn1sps/N1_Service_Provisioning_Sys tem_5.2/common/lib/mail.jar:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/se rver/lib/naming-common.jar:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/ser ver/lib/naming-resources.jar:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/c ommon/lib/postgresql.jar:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/commo n/lib/rox.jar:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/servl et.jar:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/servlets-com mon.jar:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/servlets-de fault.jar:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/servlets- invoker.jar:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/sps-com pSDK.jar:/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/server/lib/struts.jar :/opt/SUNWn1sps/N1_Service_Provisioning_System_5.2/common/lib/xerces.jar:/opt/SU NWn1sps/N1_Service_Provisioning_System_5.2/common/lib/xml-apis.jar

2006-12-22 15:03:13,064 WARN [Connection [10.0.2.3:98304]-[10.0.2.2:80001]:ErrorReader] SSH.STDERR (SshConnection.java:98) - Invoking the app...

||--|

|ID|Name|

||--|

|010010001024-0000000000000-00001-0000000005 |masterserver|

|137051083099-1165924589136-00376-1504337208 |osprovisioner|

|137051083099-1166612169104-00901-1484204905 |osprovisioner-jet|

|137051083099-1166033213201-00790-1953291407 |osprovisioner-osprovisioner|

|137051083099-1166680721157-01182-1555984346 |sunfire-v240-host-target|

||--|

bash-3.00$

Not sure why it is looking for "upgrade directories", but otherwise the CLI executes fine.

peteziua at 2007-7-7 17:55:58 > top of Java-index,Administration Tools,N1 Service Provisioning System...
# 3

I think you get a HostKeyVerification failure if the host key has changed since the last time the keys were exchange. Regardless, executing the cli directly should show the same error.

I think you're following the right path in trying to figure out which user the CLI is running as. I'd look for the execNative step XML definition that runs the CLI command and see if it specifies the userToRunAs attribute, if it does, then thats the user the CLI is running as, otherwise the default user specified in the MS config.properties file as the value of the property pe.defaultUserToRunAs is used.

Another shortcut is to edit cr_cli script and insert a line

id >> /tmp/cli_uid

into it. That will tell you the ssh uid pretty quick. Make sure that you undo this change after you're done debugging.

You can also supply ssh debug flags (-v) to get it to give you more debug output by editing the cli's config.properties as follows

net.ssh.args=-o|BatchMode yes|-v|-v|-v

hope that helps,

Aj

aj.a at 2007-7-7 17:55:58 > top of Java-index,Administration Tools,N1 Service Provisioning System...
# 4
Aj, Excellent ideas, thank you very much. I will let you know how it goes.Pete.
peteziua at 2007-7-7 17:55:58 > top of Java-index,Administration Tools,N1 Service Provisioning System...
# 5

Aj,

CLI is trying to execute as root. Is this from a lack of specifying an appropriate user to run as in the plan? My settings in MS config.properites were lacking an entry for pe.defaultUserToRunAs, so I added a valid account, restarted the MS and things still are trying to run as root.

Pete

Message was edited by:

peteziu

peteziua at 2007-7-7 17:55:58 > top of Java-index,Administration Tools,N1 Service Provisioning System...
# 6
...and still can't get Create Profile to run properly. This has been a tiresome adventure.
peteziua at 2007-7-7 17:55:58 > top of Java-index,Administration Tools,N1 Service Provisioning System...