Problem in setting ClassLoader

There's a big difference between ClassNotFoundException and NoClassDefFoundException. The first one means the class wasn't there. The second one means either that the class was there but some class it relies on wasn't, or more probably that the .class file was found in the expected location but the package or class name in the .class file didn't agree with the location. So you need to check your JAR file layout.

ejpa at 2007-7-11 23:21:55 >

But you can't change the classloader that was used to load the Python class, because it's already loaded. It's looking as though classes which a class depends on are loaded by the same class loader that loaded the original class. But I'm not an expert on this, maybe someone else has a better explanation.

ejpa at 2007-7-11 23:21:55 >

> It is possible that the getMethod method and the

> invoke method do not recognise the String class and

> the char[]-s as these are loaded by the system class

> loader and they would have to be loaded by the custom

> class loader "myloader" for them to be compatible

> with pythonInterpreterClass and execMethod.

No they wouldn't. This difficulty is not possible. It is imaginary.

> Classes loaded by different class loaders are not compatible.

That statement refers to the same class loaded by two loaders.

ejpa at 2007-7-21 19:54:09 >

Thanks for the correction ejp,

I read through part of the JVM specification, have tested some cases and class loading is now more or less clear to me, but one thing I still cannot solve though is when an object is deserialized by someone else's code (like if you pass it as an argument to a session bean and the EJB container deserializes the argument) and you need to load the class definition from a .jar file. If you could help here, I would appreciate very much. See also http://forum.java.sun.com/thread.jspa?threadID=5163767

toon_macharisa at 2007-7-21 19:54:09 >

Actually, he is using his own classloader. The problem I have is why use reflection when a simple interface works much easier and cleaner. If you do not use an interface, your goign to have ugly reflection everywhere in your program. Terribly messy and error prone.

Also, linking or not linking wont matter in this case because as soon as you try to use the class its going to get linked. So you can use either method. Only time to use the method that allows you to choose linking or not is if you are doing something special where you don't need the class to be linked yet. I have such a case where I want to just ensure all the class definitions are loaded (because they come from RMI remotely) but I don't need to use them just yet.

_dnoyeBa at 2007-7-21 19:54:09 >

I am trying to do something very similar. In my code, I am subclassing URLClassLoader and expossing the addURL() method to the public. I have done some experimentation with reflections calling this method at runtime, and adding a URL has the effect of a dynamic ClassLoader. Unfortunately the "loophole" in Reflections that allows one to call a protected method doesn't work on the Mac, so I thought of switching to subclassing instead.

I have a class, EMFMain, that loads all the Jars in a directory, creates one of these DynamicURLClassLoaders, and sets the parent to the ClassLoader.getSystemClassLoader(). The code looks like this:

/**

* This is the main driver for the Enterprise Management Framework.

*/

package com.codechimp.emf;

import java.io.File;

import java.io.FileFilter;

import java.io.FileInputStream;

import java.net.URL;

import com.codechimp.emf.DynamicURLClassLoader;

/**

* @author Michael Bauer

* @version 0.9a

*

*/

public class EMFMain {

private static File emfHome;

private static DynamicURLClassLoader dLoader = new DynamicURLClassLoader();

/**

* Main method. Loads the Spring Context XML file and starts all the

* services.

*

* @param args

*/

public static void main(String[] args) {

// Setup the EMF_HOME

String f = System.getenv("EMF_HOME");

if (f == null || f.equals("")) {

// EMF_HOME was not set, use current user.dir

f = System.getProperty("user.dir");

}

try {

emfHome = new File(f);

if (!emfHome.exists())

throw new Exception("EMF_HOME " + f + " did not exist.");

} catch (Exception e) {

System.err.println("Could not determine EMF Home: " + e);

System.exit(-1);

}

loadProperties();

loadLibraries();

try {

System.out.println("DynamicURLClassLoader:");

URL [] urls = dLoader.getURLs();

for(URL url : urls)

System.out.println("\t"+url);

EMFServer server = (EMFServer) dLoader.loadClass(

"com.codechimp.emf.EMFServer").newInstance();

Thread t = new Thread(server, "EMFServer");

t.start();

t.join();

} catch (Exception e) {

System.err

.println("An unrecoverable exception occured trying to start the EMFServerThread: "

+ e);

e.printStackTrace(System.err);

System.exit(-1);

}

}

/**

* Loads the properties tiles.

*

*/

private static void loadProperties() {

try {

File props = new File(emfHome.getAbsoluteFile()

+ System.getProperty("file.separator","/") + "conf");

FileFilter filter = new FileFilter() {

public boolean accept(File file) {

String p = file.getAbsolutePath();

return (p.toLowerCase().endsWith(".properties"));

}

};

File[] listing = props.listFiles(filter);

if (listing != null)

for (File f : listing)

System.getProperties().load(new FileInputStream(f));

} catch (Exception e) {

System.out.println("An error has occured loading the properties: "

+ e);

System.exit(-1);

}

}

/**

* Loads all the libraries in the EMF_HOME/libs directory

*

*/

private static void loadLibraries() {

try {

File libs = new File(emfHome.getAbsolutePath()

+ System.getProperty("file.separator","/") + "lib");

FileFilter filter = new FileFilter() {

public boolean accept(File file) {

String p = file.getAbsolutePath();

return (p.toLowerCase().endsWith(".jar") || p.toLowerCase()

.endsWith(".zip"));

}

};

File[] listing = libs.listFiles(filter);

for (File f : listing)

dLoader.addURL(f.toURL());

} catch (Exception e) {

System.err

.println("An error has occured trying to load the libraries: "

+ e);

System.exit(-1);

}

}

}

The EMFServer, which is in a separate JAR (I'll explain later), gets loaded along with all the other libraries. It uses the Spring Framework to do its bidding.

package com.codechimp.emf;

import org.springframework.context.ApplicationContext;

import org.springframework.context.support.ClassPathXmlApplicationContext;

public class EMFServer implements Runnable {

private static ApplicationContext appCtx;

private static boolean running = false;

public EMFServer() {

Object o = this.getClass().getClassLoader();

System.out.println("ClassLoader: "+o.getClass());

}

public void run() {

appCtx = new ClassPathXmlApplicationContext("componentContext.xml");

running = true;

while(running) {

// Do something...

}

}

}

Here is the error I am geting:

DynamicURLClassLoader:

file:/Applications/EMF/lib/emf_event.jar

file:/Applications/EMF/lib/emf_server.jar

file:/Applications/EMF/lib/spring-2.0.2.jar

Exception in thread "main" java.lang.NoClassDefFoundError: org/springframework/context/ApplicationContext

at java.lang.Class.getDeclaredConstructors0(Native Method)

at java.lang.Class.privateGetDeclaredConstructors(Class.java:2328)

at java.lang.Class.getConstructor0(Class.java:2640)

at java.lang.Class.newInstance0(Class.java:321)

at java.lang.Class.newInstance(Class.java:303)

at com.codechimp.emf.EMFMain.main(EMFMain.java:57)

My understanding is this. Since the EMFServer class is loaded by the DynamicURLClassLoader, his internally referenced ClassLoader is pointed to that one. The EMFMain's, being the class started by the JVM, is pointed to the one rerturned by ClassLoader.getSystemClassLoader(), which cannot be changed easily. I know, from much experience with JBoss, that ClassLoaders can look UP but not DOWN (I.E., Classes known by the parent can be seen by the child, but not visa versa).

It seems the error I am seeing is a result of me defining the ApplicationContext as an Class member variable. But, I am a little confussed. Is it the job of the EMFMain's classloader to resolve this class even thought I am clearly using the instance of DynamicURLClassLoader to load the EMFServer class? I was under the impression that since EMFServer was loaded by the DynamicURLClassLoader, his internal ClassLoader would point to it, and any further attempts to reconcile a Class would first go there. Is there a better, more appropriate way to do this?

mbabauera at 2007-7-21 19:54:09 >

> I am trying to do something very similar. In my

> code, I am subclassing URLClassLoader and expossing

> the addURL() method to the public. I have done some

> experimentation with reflections calling this method

> at runtime, and adding a URL has the effect of a

> dynamic ClassLoader. Unfortunately the "loophole" in

> Reflections that allows one to call a protected

> method doesn't work on the Mac, so I thought of

> switching to subclassing instead.

>

> I have a class, EMFMain, that loads all the Jars in a

> directory, creates one of these

> DynamicURLClassLoaders, and sets the parent to the

> ClassLoader.getSystemClassLoader(). The code looks

> like this:

I have not diagnosed your problem directly but I can start you out with a tip. I know lots of folks do this, but I do not like using getSystemClassLoader(). It is not very portable for code that may also use its own classloaders.

For instance, if you have code that has loaded a class by a URLClassloader, then within that class you have your code. In that case, your code will be skipping the URLClassloader and goign straight to the System Classloader, thus potentially missing some key classes.

Instead why don't you try

getClass().getClassLoader();

_dnoyeBa at 2007-7-21 19:54:09 >

> > The other solution only loads the class, but would

> > not link it (according to the API documentation)

>

> That's not 'according to the API documentation' at

> all. It doesn't say any such thing.

The API and some other back ground information combined make come to this conclusion.

According to the JVM specification, classloading occurs in 3 steps:

1. loading, which consists of finding and loading the binary data of a type in memory

2. linking, which consists of processing the binary data of a class definition in memory

3. initialization, which is the execution of the static initialisation of a class when it is first referenced to.

In the class ClassLoader:

The method loadClass(String name) is equivalent to the method loadClass(name, false), which only loads the class and does not link it. The method resolveClass(Class<?> c) links the specified class (and has a misleading name).

The method Class.forName(String name, boolean initialize, ClassLoader loader) loads the class and links it, using the given class loader. Whether the class is initialized depends on the boolean value.

A reply to _dnoyeB: yes reflection can clutter things up, but in case the guy would have to use a lot of reflection calls, he can create proxies (java.lang.reflect.Proxy). The object loaded with the custom class loader is wrapped in an InvocationHandler. The InvocationHandler and a suitable Interface are used to create the proxy. The proxy is cast to the interface and the InvocationHandler takes care of the reflection (behind the scenes).

toon_macharisa at 2007-7-21 19:54:10 >

> This is a NoClassDefFoundError, not a

> ClassNotFoundException. NCDFE is an indication that

> your runtime classpath does not, as a minimum, have

> all the classes of your compile time classpath. This

> has nothing to do with your dynamic classloading

> scheme. (unless you have a misunderstanding)

Thanks for the reply. I understand what NCDFE means, but I am more confussed about the class printout and the Class refered too via the error.

As you can see from the output, EMFServer is loaded by the DynamicURLClassLoader, which is a subclass of URLClassLoader. My understanding of ClassLoader is that a class has a reference to the ClassLoader that actually loaded him. Since I loaded the JAR in the DynamicURLClassLoader, I expected it to reference him.

In addition, the class its complaining about is org.springframework.context.support.ClassPathXmlApplicationContext, which is in the spring-2.0.2.jar. That JAR is being loaded by the DynamicURLClassLoader, so should be available in the same ClassLoader that loaded EMFServer. But, this doesn't seem to be the case, since it tosses the exception.

I am wondering if my whole approach is flawed. Perhaps I should write my own ClassLoader from the ground-up instead of subclassing. I know there are other applications that do this sort of thing, JBoss and Squrrel come to mind first, so I know its do-able. I just haven't found any "best practices" when it comes to loading JARs dynamically at Run-Time.

mbabauera at 2007-7-21 19:54:10 >

First, let me appologize if I came out sounding negative...I am just frustrated, and did not mean to direct that at anyone.

That said, I tried altering the code as you suggested:

/**

* This is the main driver for the Enterprise Management Framework.

*/

package com.codechimp.emf;

import java.io.File;

import java.io.FileFilter;

import java.io.FileInputStream;

import java.net.URL;

import com.codechimp.emf.DynamicURLClassLoader;

/**

* @author Michael Bauer

* @version 0.9a

*

*/

public class EMFMain {

private static File emfHome;

private static DynamicURLClassLoader dLoader = new DynamicURLClassLoader();

/**

* Main method. Loads the Spring Context XML file and starts all the

* services.

*

* @param args

*/

public static void main(String[] args) {

// Setup the EMF_HOME

String f = System.getenv("EMF_HOME");

if (f == null || f.equals("")) {

// EMF_HOME was not set, use current user.dir

f = System.getProperty("user.dir");

}

try {

emfHome = new File(f);

if (!emfHome.exists())

throw new Exception("EMF_HOME " + f + " did not exist.");

} catch (Exception e) {

System.err.println("Could not determine EMF Home: " + e);

System.exit(-1);

}

loadProperties();

loadLibraries();

try {

System.out.println("DynamicURLClassLoader:");

URL [] urls = dLoader.getURLs();

for(URL url : urls)

System.out.println("\t"+url);

Runnable server = (Runnable) dLoader.loadClass("com.codechimp.emf.EMFServer").newInstance();

Thread t = new Thread(server, "EMFServer");

t.start();

t.join();

} catch (Exception e) {

System.err

.println("An unrecoverable exception occured trying to start the EMFServerThread: "

+ e);

e.printStackTrace(System.err);

System.exit(-1);

}

}

/**

* Loads the properties tiles.

*

*/

private static void loadProperties() {

try {

File props = new File(emfHome.getAbsoluteFile()

+ System.getProperty("file.separator","/") + "conf");

FileFilter filter = new FileFilter() {

public boolean accept(File file) {

String p = file.getAbsolutePath();

return (p.toLowerCase().endsWith(".properties"));

}

};

File[] listing = props.listFiles(filter);

if (listing != null)

for (File f : listing)

System.getProperties().load(new FileInputStream(f));

} catch (Exception e) {

System.out.println("An error has occured loading the properties: "

+ e);

System.exit(-1);

}

}

/**

* Loads all the libraries in the EMF_HOME/libs directory

*

*/

private static void loadLibraries() {

try {

File libs = new File(emfHome.getAbsolutePath()

+ System.getProperty("file.separator","/") + "lib");

FileFilter filter = new FileFilter() {

public boolean accept(File file) {

String p = file.getAbsolutePath();

return (p.toLowerCase().endsWith(".jar") || p.toLowerCase()

.endsWith(".zip"));

}

};

File[] listing = libs.listFiles(filter);

for (File f : listing)

dLoader.addURL(f.toURL());

} catch (Exception e) {

System.err

.println("An error has occured trying to load the libraries: "

+ e);

System.exit(-1);

}

}

}

But I am still getting the same NCDFE. Also, the output still suggests that the EMFServer class is being loaded by class sun.misc.Launcher$AppClassLoader. This is my first dive into wrtting a ClassLoader of any sorts, so I am sure I am making more than one rookie mistake.

mbabauera at 2007-7-21 19:54:14 >

Well, I found the problem. When you mentioned I hadn't posted the DynamicURLClassLoader, it made me take a second look. Well, the default contructor was called the constructor URLClassLoader(new URL[0],ClassLoader.getSystemClassLoader()) in my feable attempt to force the parent loader. As was mentioned early on, this is not a good idea. As soon as I took that out, it started working just fine.

The new class looks like:

package com.codechimp.emf;

import java.io.File;

import java.io.FileFilter;

import java.net.MalformedURLException;

import java.net.URL;

import java.net.URLClassLoader;

import java.net.URLStreamHandlerFactory;

import java.util.ArrayList;

import java.util.HashMap;

public class DynamicURLClassLoader extends URLClassLoader {

public DynamicURLClassLoader() {

super(new URL[0]);

}

public DynamicURLClassLoader(URL[] urls, ClassLoader parent, URLStreamHandlerFactory factory) {

super(urls, parent, factory);

}

public DynamicURLClassLoader(URL[] urls, ClassLoader parent) {

super(urls, parent);

}

public DynamicURLClassLoader(URL[] urls) {

super(urls);

}

@Override

public void addURL(URL arg0) {

super.addURL(arg0);

}

}

The reason I did this instead of just using the URLClassLoader is that on my Mac the code I usually use doesn't work. Usually, I do something like:

URLClassLoader loader = ClassLoader.getSystemClassLoader();

Method m = URLClassLoader.getDeclaredMethod("addURL",new Class{URL.getClass()});

m.invoke(loader,new Object{<some file url>});

Strangely enough, this technique works on Windows to get around private and protected methods. Reflections apparently doesn't check the method's visibility, it just runs it. On the Mac, however, this causes an InvocationException stating the method is protected. Seems the JVM on the Mac is a little more strict in that sense.

mbabauera at 2007-7-21 19:54:15 >