When sending data (the response to the client) via the Ethernet Shield I can see in the Network Analyze program (Wireshark) all data are sent byte by byte. For that reason there are a lot of interactions between the client and the server. Are there any possibilities to send the response in one "block" over the network. Total data length is approx 300 bytes

That's a very interesting observation. In theory, yes, I think that should be possible. You'd need to look more closely to see how the library is implemented. Have you tried the Ethernet2 library? If so, does it function the same way?

A quick look at this suggests the reason for the issue is that the client send code is implemented on top of the Print class and at the bottom of the "stack" is a routine that prints a byte. And to print a string that routine gets called multiple times.

I don't know if it's possible to override the higher-level methods but it would seem worthwhile as this must be adding a fair bit of overhead. The underlying ethernet code is certainly capable of sending multiple bytes.

I took a look at this since I was using WireShark to track down another issue (with my code/Adobe Flash not the library :-)).

I can confirm what the OP posted, in my case for each byte of data sent via println 60 bytes were actually being sent on the wire--this includes for the response headers etc.

Unfortunately trying to solve this raises the whole issue of buffering etc which means it's not quite a straight fix of println. Although if the data isn't in PROGMEM then it's probably more straight-forward but that's hardly ideal.

Reason for me to care about this is the communication cost. As you have seen there are a lot of overhead. I have the Arduino stuff in my boat and connected to a mobile network and I have to pay extra if I send to much data. Unfortunately I am not a programmer so I can't contribute with coding skills. I hope you can find a solution

It's really only an interim solution and will only work if you make use of the println or print methods and supply a string.

I've only tested it with the standard WebServer example but that resulted in ~75% fewer bytes being transmitted. Even in this scenario there's still a lot of overhead as the numerals are still transmitted one byte at a time. Much more detail and analysis at the link above.

Some more observations. As every PRINT and PRINTLN generate a "segment" "on the line" I have tryed to reduce the numbers of these statement and instead put in as many chars as possible per PRINT, about 10 to 20. But there seems to be some limitation in the code or the Arduino HW because I got some hang up in the communication at some point. Very strange, I reduce the number of PRINT but more data per PRINT up to some limit

Additional information: apparently no one involved in developing the Ethernet Library has tested communications with web servers other than Apache. The library as-is will not communicate with lighttpd. Phil's modification above does allow communication, you will need to take care to render each line into a complete string before trying to println() it; multiple print() commands per line will not work.

So now I need to figure out why the Ethernet library hangs after four connection attempts at 5 second spacing, even with stop() command after each connection. At this point I know the connection idles in the FIN_WAIT state for less than a second up to 20 seconds. If another connection is attempted during this time, apparently the library makes a new one...it's limited to four. The Client::connected() function does not address this at all.

All right, I think I have it figured out. If you try to connect again before the connection has completely ended, you will start another connection. This is fine as long as you don't go above four connections. The easiest way to not deal with connection pool management is to just wait until your current connection is really closed.

The act of checking the status register actually slows down the disconnect cycle. The code below should be placed after you have finished receiving all your data:

Note that I'm using the status() function which is not exposed on the Ethernet library reference. This is a direct access to the W5100 Sn_SR status registers; all you need to know is that !client.connected() may not actually signify an ended connection. The delay(5) is necessary to give the W5100 some breathing room while it tries to terminate the connection.

Using this method, I am able to reliably download a web page about twice per second, probably a limit of the SPI and my processing routines. The time between downloads varies somewhat, but does not go over 1 second.

For ultimate reliability, you may want a timeout in the above code in case it somehow gets stuck waiting for a connection that never closes.