For small size, Oly EPL/M-x Pen body of any type. Recently EPL-5 has the most capability, largest mp sensor. Panasonic lenses and bodies are generally larger than their Oly counterparts almost as large as DSLR equipment and you won't gain much in portability. Primes for that body: the incomparable 20 f1.7 Panny, 45 1.8 or 75 1.8 Oly depending on budget and FL preference, 14 2.5 Panasonic. Zooms: Perhaps the 14-42 Panasonic X (questionable quality but tiny), 45-150 old or new Oly zoom, 9-18 Oly zoom but this might put you over budget.

If you want an eye-level built-in viewfinder, Panasonic G3 and G5 bodies, both discontinued, would be the smallest and operationally most like your DSLR. However, these are functionally bigger than the Olympus Pen bodies. When you get to the quite expensive Panny GH3 or Oly OM-D bodies, you are back up in DSLR size and, IMO, gain almost nothing over the Nikon gear you now have.

Nikon 1 system cameras are as large as the Oly bodies but the sensor is smaller, lens build looks like crap, and few lenses are available. The fact that Nikon can advertise heavily and saturate the mass market retailers like Best Buy should not sway you.

I've just made two European trips with an Oly EPL-1 body, 3 zooms, 3 primes, and tiny flash. ALL of this fit into a $50 Olympus TEN-inch long mini-backpack. I've abandoned my Nikon gear for these trips.