I started getting same error for eclipse after installing QTP on my laptop.

when I tried to run a simple java hello world program from command prompt , I got below error :

c:\Java\test>javac MyFirstApp.java
Picked up JAVA_TOOL_OPTIONS: -agentlib:jvmhook
Picked up _JAVA_OPTIONS: -Xrunjvmhook -Xbootclasspath/a:C:\PROGRA~2\HP\QUICKT~1\bin\JAVA_S~1\classes;C:\PROGRA~2\HP\QUICKT~1\bin\JAVA_S~1\classes\jasmine.jar;C:\Users\vivekrs\Documents\Raj\sev2\Tools\
QTP11\QuickTest\MSI\bin\java_shared\bin
Error occurred during initialization of VM
Could not find agent library jvmhook on the library path, with error: %1 is not a valid Win32 application

Then I removed JAVA_TOOL_OPTIONS and _JAVA_OPTIONS variables from Environments and java and Eclipse both started working fine.

you can try type javac in cmd prompt and see what error you are getting, and try to resolve that error, elipse will start working.

First of all: I've just tried it with - well - almost the configuration you have (Win8-64bit), but with already a newer java and the SR2 release of Eclipse-DSL. No issues. But I don't think the previous release could be the issue here, but if nothing helps: try downloading the SR2-release.

I know it's all stated in your post, but really make sure you're starting your 64-Bit eclipse with a 64-bit java. Also you always have to provide one of the java.exe, javaw.exe or jvm.dll in the vm-argument, if you provide the vm-argument - not just the java directory.

Also try the following: start with

-vm "<yourjavainstallation>\bin\java.exe" -consoleLog -debug

mind: it is java.exe, not javaw.exe - a console (command window) will appear - maybe the messages will help?

If all fails - try to do it with a 32-bit java and 32-bit eclipse installation. You can use the newest java - JDK 1.6 is not needed. One last point: the required java version=1.5) suggests you have an ancient JDK on your system (if the error message is correct that is). If you still have a JDK 1.4 and older on your system - try to deinstall it, if you really do not need it any more (I wouldn't find a reason what for actually).

One last thing: There is a way to make sure eclipse uses a specified jre without the vm-argument. You can actually copy the needed jre to your eclipse-installation in a sub-folder called "jre" resulting in the following structure:

eclipse
eclipse.exe
jre
bin
java.exe
...
lib
...

and start the eclipse without the vm-argument. I hope one of the points will help!