I have a GPS device that requires holding down a push button for about 4 seconds to turn on.

Ideally, I'd like to add a few non bulky components which will perform this action whenever power is connected to the device to automatically turn it on. The button cannot be held continuously as it changes modes if held for more than about 8 seconds.

If possible, I'd like to avoid a relay unless I can find a very small relay that will fit within the casing of the device.

The GPS device is 5V DC based and though I haven't measured it, I doubt the push button has more than a nominal voltage and current across it.

I've searched the forums and there are plenty of delay on and delay off relay circuits, but I've not found one that will do what I need.

Thanks for the prompt reply. I'll have to open it up and measure from the push button switch, which I won't be able to do until tomorrow.

The device is not running off a battery, but rather a DC-DC step down regulator which outputs 5vdc. I've removed the battery from the device and ideally would like to put the components in the battery housing. I have confirmed that the device works as intended without the battery and connected to the 5vdc source.

If the push button is normally open, you might get the desired result by connecting a capacitor across the switch contacts.
It would be discharged at power up, mimicking a closed switch, and then would charge eventually 'releasing' the switch.
The capacitor value would depend on the current which flows through the switch and whether it works at all would depend just what the button is connected to.

I think it is a safe starting point assumption that the switch is SPST, normally open. If so, you will be bringing out one wire from each contact no matter what, so start there. There should be no harm in trying Albert's approach first. It certainly is the bare minimum. If the device doesn't like the slow voltage ramp, or the open circuit current is so large that the cap is too large for the space (1 mA charging current could require a 2200 uF cap), then we move on to a delay circuit like a R-C-MOSFET.

The timing should be roughly proportional to the capacitor value. The value to use will have to be determined by experiment as we don't know anything about the circuit the button connects to. A sensible value to start at might be 100uF.

The current across the switch is 0.09 mA. Switch time to start the device, is 2 seconds, and it does not appear that there is any effect if the button is held. Releasing the button once the device has started and then re-applying it for 2 seconds shuts down the device.

Assuming the supply is 5V and the device counts the switch as open at about two-thirds of the supply voltage then:
(roughly) C = T / R, C = 2 / (5 / 0.09 kilo ohms), C = 36uF
Note that that value is calculated for 2 seconds (and is pretty speculative anyway). 100uF should give around 6 seconds.
Experiment is the name of the game.

The capacitor negative should be connected to the button connection which is negative.

While very little in electronics is "linear", charging a capacitor with a constant current is. Your 0.09 mA almost certainly is set by a pullup resistor, so it is not constant as the capacitor charges up. However, we can use it for a first order approximation.
ec=it
voltage times capacitance equals current times time. This falls out of the integral equation relating the current charging a capacitor to the voltage developed across it, when the current is a constant.

Voltage - assume 5 V (as above) and 1/2 half of that as the trip point for the switched circuit = 2.5 V
Capacitance - to be calculated
Current - starts at 0.09 mA
Time - 2 seconds

C = (i x t) / V
C = (0.00009 x 2) / 2.5 = 72 uF

A constant current of 90 uA will charge up a 72 uF capacitor from 0 V to 2.5 V in 2.5 seconds.

If the current is not constant, but is set by a constant voltage and a fixed resistor, the current will decrease as the capacitor charges up. This means it will take longer than 2 seconds to reach 2.5 V; or, a delay of 2 seconds will take a smaller capacitor.

After a bit of tinkering I found that 141 uF (3 x 47 uF capacitors) works reliably. I guess the current is not constant as you suggested, which is why more capacitance was needed. I don't know the exact tipping point; I was just using the 47 uF I had l lying around.

With a little tracing on the board I found locations that I could add the capacitors without fouling the casing. A little fiddly, but eventually it all works as hoped!