System.Net.Sockets.SocketException under load

We are running into client socket exhaustion issue when making a large number of sequential calls to a Web API service using System.Net.Http.HttpClient. I know that we can adjust TCP/IP socket connection timeout on OS level on the client but that
doesn't really solve the issue but rather delay when it starts happening. :-) Is there a way to have HttpClient pool connections for server-to-server communication? If not, is it something that is considered for future releases of WCF Web API?

The exact error we are getting: System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted