WS7 Forgetting JSP changes
Hello, I've been seeing a sporadic problem on my WS 7.0 deployment that I cannot reproduce consistently, but it has been happening repeatedly.
I have several configurations with one or more webapps deployed (in directories on the server rather than uploaded WAR files), and these webapps include one or more JSPs. From time to time, I will update a JSP to add some new or modified content, and when I reload the page, the changes show up as expected (my Dynamic Reload Interval is set to 60 seconds).
However, at some undetermined amount of time after making such a change (from hours to weeks later), WS7 will forget that I had updated the JSP, and revert back to a previous version of the content. If I simply "touch" the file to change the modification date on the JSP and reload the page, the updated content appears again.
This is clearly a bug, but I could not find it on SunSolve.
Is this a known issue, and when will a fix be available?
Thanks,
Bill
[1001 byte] By [
wgkorba] at [2007-11-27 3:21:26]
# 1
Hello,
I guess you are not precompiling JSP pages (By default, they won't precompile).
When the request comes to a JSP page, we check the time stamp against previous compiled JSP-generated servlet classes. If we find that the JSP page has been touch'ed since last compilation, we recompile.
In your case, you are saying that Web Server 7 is serving you correctly right after you touch the JSP page and but after sometime, for the same request, you see old content. That's not possible. This is b/c once it serves the request to particular JSP page, the JSP-generated servlets' compiled classes will be in "generated" folder under instance directory. If you still see this problem, check for the timestamp for these classes and check if their time stamp is later than the JSP page time stamp.
I can help you if you still this problem.
# 2
hi thanks for reporting this issue. we will actively try to reproduce it at our end and if it is reproducible at our end, we will definitely file a bug and let you know
# 3
Sorry if my previous response is confusing. I was just making sure that you are not precompiling jsps in your case. I was not suggesting you to precompile jsps.
# 4
No, I don't usually precompile JSPs. The next time this happens (and there will be a next time) I will compare the timestamps on the source file and the generated file.Thanks,Bill
# 5
1 question.Did you ever perform a deploy-config --force (either from CLI or GUI)?Had you performed a jsp pre-compilation on a add-webapp ?
# 6
I do not believe I have done a force (for sure not from the CLI, not sure how you do that from the GUI - I have done a pull and deploy after manually changing config files, however).
I may have done a pre-compile at some point, but certainly not on all of the webapps that have exhibited this issue.
Thanks,
Bill
# 7
OK, this happened again. I performed some server configuration this weekend, and upon deployment of the config changes, part of the site had reverted to an earlier incarnation.
My "reconfiguration" consisted of adding a new VS instance, a new listener, and requesting a server certificate for the new listener. When I deployed my configuration changes, the server reloaded all of the webapps, and that appears to be when it hit the fan.
Here are the two files that were showing their old versions:
-rwxrwxr-x+ 1 wwwwww 747 May 3 20:32 index.jsp
-rw-r--+ 1 wwwwww 195 May 3 20:31 main.jsp
As you can see, they've not changed for nearly a month.
In the server instance's generated directory for the default-webapp, I see this:
-rw-1 wwwwww 3468 May 27 08:28 index_jsp.class
-rw-1 wwwwww 2490 May 27 08:28 index_jsp.java
-rw-1 wwwwww 5810 May 27 08:28 main_jsp.class
-rw-1 wwwwww 6246 May 27 08:28 main_jsp.java
This timestamp more or less correlates with when I deployed the config changes. Looking at the .java files, it does indeed contain the appropriate content for the old versions, not the current ones.
So where did this old content even come from?
Let me know if you need any more information.
Thanks,
Bill
# 8
Questions:1. Are you using the GUI to deploy?2. If so, when you choose to deploy does the deploy dialog give 2 options (discard and pull)?3. If so, which option did you choose?
# 9
1. Yes.
2. I've seen both before, but I don't believe that I did this time.
3. Even if I did, I would have selected just deploy since I hadn't made any hand-edits to the config files.
BTW, the cert that I applied for arrived last night and I installed it (through the gui). I deployed it, then restarted the server. Sure enough, it reverted to the old content again, and I had to touch the JSPs in question to get them to recompile.
Thanks,
Bill
# 10
OK, this just happened again! This time, I for sure made the config changes and did a deploy via the GUI (and not a pull+deploy, just a deploy). When I then restarted, BANG, back to the old web site.
This is getting pretty old. Any chance you have something that I can try to alleviate this? My customer is losing his patience as this has gone on far too long.
Shall I open a new service request to move this along?
Thanks,
Bill
# 11
I am sorry that you are having this problem. There are a couple of questions I have:
a ) are the deployed web application in an NFS or shared directory? Are the generated files in a shared directory?
b ) If any on (a) is true: Is there some gross inconsistencies between the system time of the NFS server and the web server?
# 12
ws7U1 , to be released very shortly , will address CR: 6559198 which most likely will address the problem that you reported. please consider applying this upgrade and see if you can see still reproduce the problem.
# 13
I'm seeing the same issue on our web servers, but I can't find any mention of CR: 6559198 in the 7.0 update 1 release notes. See the web page below... http://docs.sun.com/app/docs/doc/820-1069/6ncp1ddac?a=viewWhere can I get more information on CR: 6559198?
# 14
i have already asked our docs team to find out how this got left out in the release notes. we very much regret for the issue that u r running into . let us see if the problem u r describing is actually addressed by this CR: 6559198. please try out ws7u1 and let us know
# 15
We will go for the update and report back, but can you please respond with the details of CR: 6559198. I'm very curious as to what exactly it mentions.
# 16
Could you confirm the following for me?
1. Does the deploy dialog have 2 radio button?
2. When you said "just a deploy" could you tell me what were the words in the choice exactly? Does the inline help for the choice say this, "Discard changes to the node instances and deploy the current saved configuration to all nodes. All node instance changes will be lost."?
# 17
It says this:
Deployment Pending
The configuration nexstarnetwork.com has changed locally.
Click on "Deploy..." to propagate the changes to all instances
And then I click on the "Deploy..." button.
The only time I've seen the radio buttons is when I have made manual changes to the config files.
So are you suggesting that I download and install "Web Server 7.0 Update 1 Technology Preview" that is on the Web Servers download page? If that isn't production quality yet, I don't want to install it on my server where I am running several webs sites in addition to the ones that are having this problem.
When will ws7u1 officially be released? Will there be an easy upgrade process (e.g., with the SPs on the old servers, it was possible to just run the installer, point it at the existing install location, it updated the files that had changed, and then all I had to do was restart each instance)?
Thanks,
Bill
# 18
Oh, and no, this is not on an NFS-mounted filesystem - the files and the server are all on the same physical system.Thanks,Bill
# 19
And lastly, I, too, would like to see the details for bug ID 6559198.Thanks,Bill
# 20
Ah, I found it on SunSolve:
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6559198-1
However, I am skeptical, as I highly doubt that my old and new JSPs were the exact same content length. If that is the case, I should be able to edit the file, add a single blank line, then restart the server and I will never see the problem again, yes?
I just tried that, then went into the admin GUI, went into the web applications screen, clicked the checkbox and clicked the disable button for a webapp that hasn't been having this problem, then clicked the checkbox again and clicked the enable button, then clicked the pending deployment link, and deployed the change (and I did not have radio buttons, just the message detailed in my previous post).
BANG! Right back to the old content until I did a "touch" on each of the three JSPs.
Thanks,
Bill
# 21
No. I was just trying to understand the problem scenario.
Let me tell you the steps that I tried to simulate the issue.
1. Create a configuration
2. Create an instance of the configuration
3. Add web application (webapps-simple.war) with uri '/simple' and with pre-compilation
4. Deploy the configuration
5. Start the instance
6. Access the uri '/simple'.
7. Now modify a JSP of the web application on the instance.
8. Refresh the page (Notice that the changes show up on the browser)
9. Now make any modification on the configuration.
10. Deploy the configuration
11. Notice the deploy dialog has 2 options. Choosing either of them reverts the changes made to the JSP.
12. Add another web application (webapps-simple.war) with uri '/testsimple' but without pre-compilation
13. Deploy the configuration
14. Start the instance
15. Access the uri '/testsimple'.
16. Now modify a JSP of the web application on the instance.
17. Refresh the page (Notice that the changes show up on the browser)
18. Now make any modification on the configuration.
19. Deploy the configuration
20. Notice the deploy dialog has 2 options. Choosing either of them reverts the changes made to the JSP.
I am not able to effect the reverting of the changes to the JSP if I modify the JSP on the web application source project and re-deploy the web application using 'Add Web Application' again and a simple 'deploy configuration' operation.
The behaviour you are facing doesn't seem to be any of the above.
Could you provide me with the steps that you followed (and please highlight if the steps differ from any of the above steps)?
I know that you used directory deployment instead of .war files, but I am not sure about the sequence of steps that lead to the issue.
# 22
Did you try by setting the dynamic load interval back to the default (-1)? If the applications are in directories instead of war files, it should not be necessary.I am curious to know if that has something to do with it. Please, try that and report back.
# 23
I changed it to -1, deployed, and when the deployment completed, BANG, back to the old content (until I "touched" the JSPs in question).Bill
# 24
Please take a look at the steps that I followed, to reproduce the bug, and tell me if the steps that you took were different. If so, please provide the steps that you performed.
Also, I tried with both directory deployment and war file deployment on both 7.0 and 7.0u1 and was not able to reproduce the problem in both.
Also compile and execute the following code after every configuration modification and before deploying the configuration:
import java.io.*;
import java.util.HashMap;
import com.sun.web.admin.AdminEnvironment;
import com.sun.web.admin.exceptions.AdminException;
import com.sun.web.admin.configlib.ConfigTOC;
public class CheckConfigTimestamps {
public static void main(String[] args) throws Exception {
if (args.length != 1)
usage();
ConfigTOC toc = new ConfigTOC(args[0], AdminEnvironment.SERVER_ENVIRONMENT);
toc.loadTOC();
HashMap changes = toc.getConfigChanges();
if (changes != null)
System.out.println("The files that are changed are as follows: "+changes);
else
System.out.println("There are no changes in the specified configuration.");
}
public static void usage() {
System.out.println("Usage:\n\tjava -Dcom.sun.web.installRoot=<ws70_install_dir> -Dcom.sun.web.instanceRoot=<parent_dir_of_https-*> -cp <ws70_install_dir>/lib/webserv-admin.jar:<ws70_install_dir>/lib/webserv-admin-shared.jar:<ws70_install_dir>/lib/webserv-rt.jar:. CheckConfigTimestamps <config_name>");
System.exit(1);
}
}
After compiling the above code execute it as follows:
$ cd <ws70_install_root>/lib
$ <jdk_1.5.0_09_dir>/bin/java -Dcom.sun.web.installRoot=<ws70_install_dir> -Dcom.sun.web.instanceRoot=<parent_dir_of_https-*> -cp webserv-admin.jar:webserv-admin-shared.jar:webserv-rt.jar:<compiled_class_file_dir> CheckConfigTimestamps <config_name>
# 25
Hi Guys,
Weve the same issue at our production environment (Win 2003 + WS7 + JDK 1.5) and maybe our workaround can help you.
After deploy a new version of our WebApp, we immediately clean the classcache manually and it solves our issue.
Go to C:\iPlanet\https-instancename\generated\servername\org\apache\jsp\web\JSP and delete all files. It will force your web server recompile all JSP with the new version.
Remember make backup before to try it.
Ive a question:
In the old versions of iPlanet has a function to clear the ClassCache on GUI. Does this function exist on WS7.0?
Thanks
Ges7a at 2007-7-21 20:44:49 >
# 26
Could you try the following to see if it solves your problem?
$ rm -rf .../admin-server/config-store/<configname>/generated/<vsid>/<web appuri>
Execute the above command in the administration server m/c and see if it solves your problem.
BTW, if you are using directory deployment then do not use --precompile-jsp option for the add-webapp command. Instead use bin/jspc.
Also, please go ahead and raise a service request with Sun support.
# 27
After looking over your list of steps to try to reproduce the problem, I realized that the only webapp I'm consistently seeing this on is the default webapp (i.e., I created a WEB-INF under the docroot and dropped a web.xml there, and then have JSPs that live within the docroot). Some interesting things that might make this easier for you to reproduce:
1. The deployment descriptor is based on the web-app_2_3.dtd DTD as these JSPs are using some J2EE 1.3 features and have not been updated to 1.4 semantics yet - hence, the entire default webapp is forced to 1.3 semantics.
2. The following JAR files are in the docroot/WEB-INF/lib:
activation.jar
base64lib.jar
jca1.0.jar
jdo-1.0.2.jar
jstl.jar
jta-spec1_0_1.jar
kodo-jdo-runtime.jar
mail.jar
mx4j-jmx.jar
mx4j-tools.jar
oracle-jdbc-10.1.0.2.jar
standard.jar
The Kodo JDO JARs are version 3.4.1 (from Solarmetric/BEA).
3. In sun-web.xml, I specify "<class-loader delegate="false"/>" so that the above JAR files are used in preference to any that the server may provide.
4. Because this is the default webapp, there was no way that I know of to enable JSP precompile.
I will try compiling the code that you posted and use it as you described.
While the purging of the generated code might fix things for now, it's treating the symptom rather than the illness, and I'd like to help you find and fix this bug, rather than just fix my problem for now (and I'm confident that were I to do so, it would rear its ugly head again at some point in the future).
I will also open a service request with Sun support.
Thanks,
Bill
# 28
I do beleive the correct bug ID you want is CR 6554691
As noted in some of this thread the issue is coming from the config-store/generated directroy.
At some point you did a pull and deploy. During the pull operation some of the web app JSPs get placed in config-store/generated
so now when you do any deployments (config-webapp), what ever was in there from the LAST pull and deploy will get pushed out.
The fix is in 7.0u1 (that is what we are told) so look for that build when it goes out for production use.
Cheers
Robert
# 29
Also should have told you - when you do a web-app deployment do NOT use the pre-compile option. This will also update the /config-store/generated directory. This too should be fixed in update 1
# 30
The problem has been identified. The culprit is pull-config.
When a pull-config was performed it brought in all the class files from the <instancedir>/generated directory into the config-store. Since, the modified JSP was not yet accessed these class files reflect the older version of the JSP. And now pull-config has pulled these old JSP class files into the config-store. A subsequent deploy-config would push these old class files again to the instance overwriting the class files in <instancedir>/generated.
This problem has been fixed in WS 7.0U1, due for release in approximately a couple of weeks.
The precompile-jsp problem indicated by Robert will be fixed in WS 7.0u2 release.
For now you can go ahead and delete the generated/* in the config-store and deploy.
NOTE: If you use the pull+deploy of the GUI you will have to delete the generate/* in the config-store after that and perform a deploy again.
IMPORTANT: If the only modifications you are performing on the instance are the JSP/webapp changes, then you do not need to do a pull-config since you are using directory deployment. This will completely avoid the problem that you are facing.
# 31
Excellent! I'm glad to hear that the problem has been resolved.I will clean up my generated artifacts and re-deploy, and I will be sure to install u1 as soon as it is released.Thanks so much for getting to the bottom of this for me.
# 32
WS 7.0u1 has been released.You can find it at http://www.sun.com/download/products.xml?id=467713d6
