I'm working on an application that will run on a phone where the phone
will be a station on a private Wi-Fi network. The phone will be a
station, not an access point, and the private Wi-Fi network does not
route to the Internet. My application needs to communicate with
servers on the Internet as well as devices on the local Wi-Fi network,
so it needs to have connections on both networks at the same time.
I've been trying to figure out how to do this.

I've been trying the technique described in the discussion on the
Google Android developers group titled "Can
Android 2.X connect to 3G and Wifi data networks simultaneously?", but
it is not working well. What I find is that, when I enable the
cellular network by calling
ConnectivityManager.setNetworkPreference(ConnectivityManager.TYPE_MOBILE),
any sockets I have open on the Wi-Fi network are closed. I haven't
tried it, but I suspect the same thing will happen to sockets on the
cellular network when I switch back to Wi-Fi.

Another problem is that,
these calls operate on a global level, changing the network settings
for the entire phone, not just the application. Switching the network
set up globally like this will interfere with any other app that
happens to be running on the phone. Even after my application exits,
the phone continues to run with the last network configuration it set.

I'm looking for a way to have connections open on both the cellular
data and Wi-Fi networks at the same time, and without interfering with
other applications running on the phone.

1.Does anyone know how to do
this?
2.Does anyone know if this is possible?

For the second approach it is specifically stated it works with Android 2.2, no idea if this works in actual versions as well. However as far as I found out, enableHIPRI is more or less a hidden network setting, so I would prefer the first method if possible.

If you can't make it using API calls and if you are willing to get your hands dirty with the lower level, some linux knowledge may help.
Basically what you have to do is to bring up both interfaces and have the default route set on the 3g interface.
You will have to use system commands with root privileges for this kind of task.
The reason for the close sockets is probably the interface that goes down and up again because of the API call.

With Android API what you can do at most is just turn on WiFi in hope that the device will switch to it and turn WiFi off to make the device switch to 3G (if it's there, the APN is correct etc.).

Anything else is not guaranteed to work. E.g. setting preferred connectivity type doesn't guarantee that the device will switch to that type.

The usual behavior is that as soon as WiFi becomes available, the device will have both 3G and WiFi on for a short while (3-5 secs) and then turn off 3G. As soon as WiFi is turned off by the user or your app, and the device attempts to connect to the Internet, it will turn on 3G after a short while.

Starting with Android 2.3 you can't event disable/enable 3G anymore. One used to spoil/restore APN settings to enable/disable 3G, but starting with 4.0 you can't change APN settings programmatically.

The 3G and Wifi data networks can not connect at the same time, but 3G and Wifi can be connected simultaneously if the operator support and the android framework also change for the operator.
Now some operators already have this feature but some can not.

Using both network at a time is not possible in Android App at present but you can do this in PC.
Connect one from your WIFI router and one from LAN or USB Internet Stick.
So create APp and use on PC. If you do not know how to run Android on PC than google it.

If I am not mistaken, if both Wi-Fi and 3G/4G is available, Wi-Fi will take the precedence.

Only when Wi-Fi is not available, it will switch to 3G/4G. Of course this provided both are switched on at the same time.

Instead of using a private wifi thus limiting your choice you may consider having a internet facing VPN, so that the device will connect to the VPN from 3G/4G and than subsequently access the data if security is your concern for using private wifi,.

And VPN clients are inbuilt in Android and there are also third party clients available.