I am using 100.14.19 (tried 169.04 too) in Gentoo Linux.
I'm developing a Java application using the Java3D framework. Most things work great with OpenGL, including Cg/GLSL shaders.

In Java3D I can query the drivers (through the Canvas3D class) for driver capabilities. One of the properties is sceneAntialiasingNumPasses. If the value is 1 it means that multisamping hardware anti-aliasing is supported.
If it's 8 (in my case) it means that Java3D is doing 8 passes in the accumulation buffer, which is slow.

I know that my 7600 GT supports multisamping hardware anti-aliasing, so why does the driver say it's not supported? And how can I enable it?

When you say "driver say it's not supported" do you mean when polling it through Canvas3D? I assume thats the info you posted, and according to that its saying AA is available (sceneAntialiasingAvailable=true). Any good 3d interface I assume would not force high-end settings on the programmer. Try looking for a function in the API for setting the AA level (its probably set to 0 by default). Also, you should check the nVidia Display Settings program to see if AA is being forced to off (make sure Override Application Settings is NOT checked). I've never used Java3D before, so this is all the help I can offer.

Edit: To me it seems more likely a Java3D problem than a nVidia problem. I found this page that mentions full scene AA being disabled by default.

Yes, I meant by "driver say it's not supported" that when polling the Canvas3D properties it says that hardware AA is not supported.
If it were supported then sceneAntialiasingNumPasses would be 1 instead of 8.

Canvas3D / Java3D gets its properties from the driver/opengl/Xorg, not sure which one but still an external factor.

I can enable AA with the nvidia-settings utility, but that results in basically the same low framerate.

The Java3D applet is 512x512 which should be easy for my GF 7600GT. Running in windowed mode.

OK, but the way those properties are used still makes no sense to me, oh well. This doesn't mean the problem is not Java or Java3D. What versions of Java and Java3D are you using? Also, run the command glxinfo | grep multisample, if you find GLX_ARB_multisample and GL_ARB_multisample in the output then the problem is likely in Java or Java3D.

The reason forcing AA on in the nVidia Display Settings makes it run so slow is because not only is the window being forced to AA, but so is the rest of your desktop (since you're running Compiz-Fusion). The forced AA might run better if you turn off Compiz while running the program. Also, just a random crazy idea, try turning off Compiz and running the program (no forced AA) and see what happens.

Might also see if Mesa is interfering with nVidia's OpenGL libraries, though since the properties are reporting it as nVidia this probably isn't the case.

I am using Java 1.6.0_02 and Java3D 1.5.2-pre (tried various Java3D 1.5.0+ versions).

With Compiz off, a very basic desktop environment, with forced AA off, I get exactly the same results in Java3D. When using Java3D's scene antialiasing it's faster because Compiz isn't hogging the GPU, but it's not as fast I would expect it to be.

One last note: when I'm forcing AA with nvidia-settings, I only see AA results when using 8x/16x. No AA at all when using every lower setting there is.

I'm just trying to determine where this wierdness comes from, whether it's the nVidia drivers, Java3D or this particular combination.

Correct me if I'm wrong, but I should be able to AA 8x a 512x512 window without any trouble at all on a 7600GT right? Even when using a relatively heavy window manager like Compiz Fusion.

I would place my money on Java3D being the culprit as the graphics card's abilities seem to be reported properly outside of Java. I'll poll my systems capabilities from Java3D if I get a chance. If I have the same problem then it's likely Java3D. I was also taking a quick look at the Java3D source to see how it's determining the graphics capabilities, haven't found anything yet.