Introduction

In NovellÃ‚Â® ZENworksÃ‚Â® 6 Desktop Management and above, Novell includes an architectural element
known as the ZENworks Middle Tier Server. The middle-tier server is a Web server with
ZENworks Web components built on Apache or Internet Information Services (IIS) that
provide functions to the workstation agents, allowing them to deliver ZENworks Desktop
Management features.
When a ZENworks Middle Tier Server is part of the infrastructure, many workstation
agents might be directed to the same middle-tier server, making it a resource that needs
to be understood and managed properly in order to provide scalability and availability to
all of the clients in the organization.
This paper discusses the expectations that can be placed on a ZENworks Middle Tier
Server. It also discusses methods for enhancing a middle-tier server configuration to
provide greater scalability and availability.

Scalability versus Performance

Scalability is the ability to increase processing by
adding more resources. Performance refers to the
response times presented under a typical load.
Increasing system scalability does not necessarily
increase system performance. The goal of scalability
is to provide constant, acceptable performance
even as the load increases. Scaling can be achieved
by adding additional hardware (such as CPUs and
memory) to the server (known as scaling up), or
by adding additional servers to a server farm to
share the load (known as scaling out). It is vital to
understand the ramifications of implementing both
scaling methods so that you can be prepared to
choose the one that best meets your needs and to
address the bottlenecks that might occur in
a middle-tier implementation. The following
pages discuss these issues and the best ways to
improve scalability.

Scalability versus Availability

Availability is the ability for a resource to be
accessed. High availability is achieved when the
resource is always available and users can utilize
it at any time. Although providing a greater scale
can provide some perceived increases in availability,
a potential single point of failure still exists if
resources are stored on only a single server.
Availability can be achieved only through the
use of additional resources such as server farms.
This document specifies the methods that can be
used to provide high availability to your middletier
architecture.

Middle Tier functionality

In order to understand the potential bottlenecks
and the best scaling and availability architecture,
it is important to understand how the middle-tier
server interacts with the agents on the devices
and with Novell eDirectoryÃ¢â€žÂ¢ services.

Agent Requests to the Middle-tier Server

The ZENworks Middle Tier Server provides a support
function for the ZENworks Desktop Management
agents running on the user workstation. These agents
must authenticate to eDirectory and retrieve
policies and applications for both the workstation
and the user. The Desktop Management agent
accomplishes this by sending XML requests for
information to the middle-tier server. In return,
the middle-tier Web components retrieve the
informationÃ¢â‚¬â€either from eDirectory or from the
file systemÃ¢â‚¬â€and return it to the workstation
agents either through XML or with WebDav or
HTTP file transmissions.

Agent File Transfers Through the Middle-tier Server

The agent resolves file references using one of
three alternative mediums:

The Novell ClientÃ¢â€žÂ¢, if it is present on the workstation (not required for ZENworks)

The Microsoft* Client, if a drive mapping or an accessible UNC share is referenced

The ZENworks Middle Tier Server

As shown in Connection 2 in the figure, when the
Desktop Management agent goes to retrieve a file
from the network, it uses the client first if the file
reference is a mapped drive or an accessible UNC
share. If the file reference is not a mapped drive
or if the UNC share is inaccessible, the Desktop
Management agent requests a file and passes the
UNC to the middle-tier server over Connection 1.
The middle-tier server will then use the UNC path
to retrieve the file and send it back to the
workstation over Connection 1.
If administrators want to provide MSI-based
applications to users who do not have the ability
to connect directly to the file server holding the
MSI file, they must administer the MSI application
object as force-cache. The force-cache flag causes
the Novell Application LauncherÃ¢â€žÂ¢ to immediately
request the MSI files through the middle-tier
server and to store the retrieved MSI files in the
local workstation cache. When the application is
launched, the Microsoft Installer process is called,
passing the local path to the MSI files in the cache.

Middle-tier Connectivity to eDirectory

The following figure represents communication
conduits that can be created during normal activity
among a Desktop Management agent, the middletier
server and eDirectory.

When a Desktop Management agent communicates
to the middle-tier server, it opens an HTTP
connection to the Web server. This is a standard
connection that is terminated when the
communication between the workstation and
server is complete; however, to provide continuity
between HTTP connections, the Web server
creates a session cookie that is subsequently
used between the client and server. The session
terminates when no communication has occurred
between the workstation and the server, based
on configured parameters on the Web server
(traditionally 15 minutes). If the user logs out
of eDirectory, the middle-tier server terminates
the session.

When the middle-tier server needs to communicate
with eDirectory to authenticate the user and
workstation, it creates a connection between
the middle-tier server and the eDirectory server.
Because the cost of creating this connection is
high, the middle-tier server maintains it for the
length of the session rather than of the HTTP
connection. When the session is terminated,
either because it timed out or because the user
explicitly logged off, the eDirectory connection
is put into an available pool for use by another
user. If the connection is not used after five
additional minutes, it closes.

Middle Tier Performance

There are several factors that contribute to what
scale is achieved on a middle-tier server. The factors
having the greatest impact include the following:

Speed and number of CPU processors on the middle-tier server

Connectivity speed between the middle-tier server and eDirectory

Amount of data to push to each user and workstation, particularly force-cache applications and policies

Amount of RAM available on the middle-tier server

Staggered login times and launcher refresh intervals

Considering these factors, you can approach the
architecture for middle-tier installation by first
improving the hardware speeds on the middle-tier
server. When the middle-tier server is no longer
experiencing high utilization, the next bottleneck
is the connection between the middle-tier server
and the eDirectory server. This can be addressed
with a higher-speed connection between these
two machines.
Additionally, IIS and Apache can provide
a significant improvement in performance if
configuration modifications are applied. It is
recommended, to improve performance, to modify
the ThreadsPerChild configuration in the Apache
adminserv.conf file. Additionally, improvements in
IIS can be accomplished by following the suggestions
from a Microsoft published whitepaper titled
Ã¢â‚¬Å“The Art and Science of Web Server Tuning with
Internet Information Services 5.0Ã¯Â¿Â½? by George
Reilly, published March 9, 2001

After an initial application distribution, day-today
performance normally consists of routine traffic
from the Application Launcher through the middletier
server to eDirectory as ZENworks checks for
updates to its currently installed application base.
Three critical tests were performed on middletier
and back-end servers with this configuration,
with ZENworks 6.5 SP1 installed. ZENworks 6.5 SP1
represents almost a 20% performance improvement
over previous releases. The following information
describes the results of those tests.

Performance Test 1
The first performance test was an authentication-only test of workstations authenticating simultaneously
to a single middle-tier server. The results of this test on the supported server platforms are shown in the
table below:

OPERATING SYSTEM

1 CLIENT

240 CLIENTS

460 CLIENTS

NetWare 6

5 seconds

36 seconds

51 seconds

Windows 2000

9 seconds

50 seconds

48 seconds

NetWare 6.5

6 seconds

50 seconds

50 seconds

Windows Server 2003

7 seconds

26 seconds

50 seconds

Performance Test 2
The second test shows the results of an Application Launcher non-initial refreshes of 40 applications:

OPERATING SYSTEM

1 CLIENT

240 CLIENTS

460 CLIENTS

NetWare 6

7 seconds

1 minute, 14 seconds

2 minutes, 18 seconds

Windows 2000

21 seconds

1 minute, 10 seconds

1 minute, 6 seconds

NetWare 6.5

6 seconds

28 seconds

1 minute, 5 seconds

Windows Server 2003

6 seconds

1 minute, 2 seconds

1 minute, 3 seconds

Applications used for this distribution test ranged in size from 2 MB to 17 MB. Applications were randomly
associated to containers, groups and users.
NOTE: It is uncommon for 460 users to log in simultaneously every day and then for each to trigger the
distribution of 40 applications, therefore individual performance may be better than shown. Initial refreshes
will take longer in order to fill the data cached on the device.

Performance Test 3
The third test shows the results of an Application Launcher non-initial refreshes of 100 applications:

OPERATING SYSTEM

1 CLIENT

240 CLIENTS

460 CLIENTS

NetWare 6

13 seconds

3 minutes, 45 seconds

7 minutes, 18 seconds

Windows 2000

46 seconds

3 minutes, 17 seconds

4 minutes, 21 seconds

NetWare 6.5

10 seconds

1 minute, 26 seconds

3 minutes, 32 seconds

Windows Server 2003

13 seconds

2 minutes, 53 seconds

2 minutes, 35 seconds

Applications used for this distribution test ranged in size from 2 MB to 300 MB. Applications were
randomly associated with containers, groups and users.

NOTE: It is uncommon for 460 users to log in simultaneously every day and then for each to trigger the
distribution of 100 applications, therefore individual performance may be better than shown. Initial refreshes
will take longer in order to fill the data cached on the device.

A Customer Environment Tuned for Performance

The following diagram represents the ZENworks Desktop Management architecture used by a Novell customer for a large, Windows-centric production environment.

The details of this setup are as follows:

Six ZENworks Middle Tier Servers are installed on Windows 2000 servers, all located in a Microsoft load-balanced cluster. The cluster is configured to allow 900-1200 concurrent connections per middle-tier server. The number of connections varies depending on the size of the data set being transferred. (Each server is an HP/Compaq* DL360 with a dual Xeon 4.0 GHz processor, 2 GB RAM and a single 1-GB network card.)

Three Windows 2000 servers have Novell eDirectory and the ZENworks Desktop Management Server installed. (Each server is an HP/Compaq* DL360 with a dual Xeon 4.0 GHz processor, 2 GB RAM and a single 1-GB network card.)

Middle-tier servers 1 and 2 point to eDirectory Server A. Middle-tier server 1 fails over to eDirectory Server B and then to eDirectory Server C. Middle-tier server 2 fails over to eDirectory Server C and then to eDirectory Server B. Middle-tier server 3 fails over to eDirectory Server A and then to eDirectory Server C.

Desktop Management agents receive a DNS name for the middle-tier server from a local DHCP server. At headquarters, the middle-tier server DNS name points to the IP address that is assigned to the Microsoft cluster.

Although this round-robin DNS addressing
configuration works, we recommend using an L4
switch. For more information, see the sections on
Ã¢â‚¬Å“Round-robin DNS AddressingÃ¯Â¿Â½? and Ã¢â‚¬Å“Web Switching.Ã¯Â¿Â½?

Summary of Expected Performance

The day-to-day performance of ZENworks 6.5
Desktop Management (that is, the performance
after an initial application distribution) depends
on the configuration of your network environment
(for example, the hardware you are running and
the amount of data being pushed through the
network). However, given a configuration similar
to the one described in this paper, you can expect
each ZENworks Middle Tier Server to handle 900 to
1200 concurrent connections with good performance.

Windows Server TCP/IP tuning

Tuning the TCP/IP configuration on a Windows Middle Tier is needed if a large amount of workstations will be connecting through this Middle Tier server. In the Microsoft Windows Server 2003 TCP/IP Implementation Details there is some interesting information that can be used to optimize the TCP/IP configuration for the Middle Tier and the back-end eDirectory servers.

When a TCP connection is closed, the socket-pair is placed into a state known as TIME-WAIT. This is done so that a new connection does not use the same protocol, source IP address, destination IP address, source port, and destination port until enough time has passed to ensure that any segments that may have been misrouted or delayed are not delivered unexpectedly. RFC 793 specifies the length of time that the socket-pair should not be reused as two maximum segment lifetimes (2 MSL), or four minutes. This is the default setting for Windows Server 2003 TCP/IP. However, with this default setting, some network applications that perform many outbound connections in a short time may use up all available ports before the ports can be recycled.
Windows Server 2003 TCP/IP offers two methods of controlling this behavior. First, the TcpTimedWaitDelay registry parameter can be used to alter this value. Windows Server 2003 TCP/IP allows it to be set as low as 30 seconds, which should not cause problems in most environments. Second, the number of user-accessible ephemeral ports that can be used to source outbound connections is configurable using the MaxUserPort registry parameter. By default, when an application requests any socket from the system to use for an outbound call, a port between the values of 1024 and 5000 is supplied. The MaxUserPort parameter can be used to set the value of the uppermost port that the administrator chooses to allow for outbound connections. For instance, setting this value to 10,000 (decimal) would make approximately 9000 user ports available for outbound connections.

For servers with a large amount of workstations connected use the following settings:

MaxUserPort
- Use the regedit command, access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey, and create a new REG_DWORD value named MaxUserPort
- Set this value to decimal 32768 (Hex 0x00008000).

TcpTimedWaitDelay
- Use the regedit command, access the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\TCPIP\Parameters registry subkey, and create a new REG_DWORD value named TcpTimedWaitDelay
- Set the value to decimal 30 (Hex 0x0000001e). This value sets the wait time to 30 seconds.

After adding these settings to the registry the server will need to be restarted.

Increasing Middle Tier Scalability and Availability

In addition to tuning your hardware and software
parameters to provide the best performance of your
middle-tier server in your environment, there are
additional configurations and architectures that you
can use to increase its scalability and availability.

Delivering Data through File Servers

The load on an individual middle-tier server can be
greatly reduced if applications and other required
data can be retrieved through a local file server.
This option can be accomplished by first configuring
a mapped drive on the local workstation or
referencing files via a UNC share to a local server.
Then the application objects delivered to that
workstation or workstation users can reference
the mapped drive or share in order to retrieve
the application files and other data.
This option is available only to those who have
a LAN connection. Workstations that retrieve
information through the Internet but are not
connected to the LAN must receive their distributions
through the middle-tier server.
It is advisable to use the randomizing features
of the Novell Application Launcher to direct the
workstations to connect at random intervals during
a specified time frame. This precaution spreads
the number of users connecting and retrieving
files across the defined time frame rather than
allowing all users to connect at the same time
(when the workday begins, for example).

Multiple Web Servers

Performance increases when fewer users are
sent to a single middle-tier server. This can be
accomplished by directing Desktop Management
agents on different workstations to different
middle-tier servers. When the agent is installed,
a registry key is set on the workstation to maintain
the DNS/IP address of the middle-tier server to
be contacted by the agent. Using the installation
parameters for various agent installations, you can
set the middle-tier server address for different
sets of users. Even so, the application objects
accessed by various middle-tier servers can still
refer to the same storage server.

Round-robin DNS Addressing

If you want to enhance the availability of the
middle-tier servers within your environment,
you can introduce more middle-tier servers into
the network and use DNS addressing techniques to
provide a set of several IP addresses corresponding
to one DNS name. Using this addressing method,
when a workstation requests the IP address of a
given DNS name, the DNS server returns several
different IP addresses. This spreads the load across
multiple middle-tier servers. When a workstation
is successfully connected to a middle-tier server,
it uses that middle-tier server until the session is
terminated. (See Ã¢â‚¬Å“Connection 2Ã¯Â¿Â½? on page 6.)

Round-robin DNS addressing does not detect
a disabled server within the group of addresses.
This limitation introduces the potential for
degradation in system performance because
the Desktop Management agent can continually
attempt to establish connections to a disabled
server before attempting to connect to the next
server on the list.

Web Switching

Another technique for obtaining increased scalability
and availability is using an L4 (or higher) switch.
You can administer such a switch to provide access
to several servers that are referenced by the same
IP address. This way, your agents can be given the
same IP address yet can receive the benefits of
multiple middle-tier servers. This configuration
provides all of the benefits of round-robin DNS
addressing described above. A load-balancing
switch is aware of the loads of each of the
individual servers and can use that information
can also quickly detect a disabled device and
then route requests away from it.

to distribute requests. Load-balancing switches
If you decide to use a load-balancing switch,
select one that allows the agent to establish and
maintain a session with the middle-tier server.

Conclusion

ZENworks Middle Tier Servers can be scaled out or up
to provide services to a larger audience. Even with
server hardware improvements, administrators must
configure the system properly in order to provide the
greatest potential performance for their users. Administrators
should be aware of the following items:

Ã¢â‚¬Â¢ Connectivity speed between middle-tier servers
and eDirectory. Separating these server
functions among different servers provides
maximum support.
Ã¢â‚¬Â¢ The amount of data being pushed to each
user and workstation through the middle tier,
particularly force-cache applications and
policies. Using other file servers can off-load
the middle tier and improve performance
at login.
Ã¢â‚¬Â¢ Staggered login times and launcher refresh
intervals. Better performance is possible if
users do not simultaneously log in and refresh
their agents.
Using these guidelines, the ZENworks Middle
Tier Server can support many users with a robust
and positive performance experience.