The other day I was chasing some lock contention issue with a 64-bit application running on Solaris 10 Update 1; and stumbled with an unexpected dbx crash. My intention is to find the hot locks with the help of dbx's threads, syncs, sync -info <address> commands. (Of course, most of this information can be obtained with modular debugger, mdb, too - but that's a different story). So when I attached the running process to dbx, it crashed, while loading some of the libraries with the following error:

As the error message indicates, the problem is with the alignment of 64 bits (in this case) on 64-bit boundary of the object being loaded. From the bug report dbx dumped core when hitting a misaligned load in libdl.so.1, it appears that elfsign is responsible for the misaligned section headers of Solaris 10's 64-bit libraries. Since then Sun fixed the elfsign, and released majority of properly aligned 64-bit libraries as patches through sunsolve web site. So, if the system running Solaris 10 (or later) is not patched with the fixed libraries, it is still possible to see dbx crashes. This is because 64-bit dbx expects the libraries to be aligned properly on their boundaries -- it just crashes when it encounters any misaligned libraries.

Make this script executable with chmod +x <script>.pl command, and run it directly ie., ./<script>.pl (doesn't need root privileges)

Fix to Sun Studio 11's 64-bit dbx

If the above perl script returns some library names as output, it is very likely that the existing 64-bit dbx may not work on Solaris 10 (and later). Chris Quenelle of Sun Microsystems, posted a fixed 64-bit Studio 11 dbx binary, that doesn't crash even if it encounters misaligned elf section headers, in his blog. The binary can be downloaded from: http://mediacast.sun.com/share/quenelle/dbx.ss11.align.fix.gz.

Note that this is an unsupported binary. Do not copy it over the existing Studio 11 64-bit dbx. Just place it anywhere else, and make it an executable. Then simply attach the running process to the new 64-bit dbx binary.

Dr. Edsger W. Dijkstra, who coined the word Structured Programming, doesn't need a special introduction, to all those people in the field of computing. Shortest Path algorithm, semaphores solution to the classic Dining Philosophers problem, notion of invariants, pre- and post-conditions for the correctness of programs, strong case against GO TO statement are some of his contributions to Computer Science. Dr. Dijkstra was also a principal contributor to the development of the ALGOL programming language.

Dr. Dijkstra was known for the habit of writing his opinions on programming/computing (computer science, in general) in a series of manuscripts called EWDs. Dr. Dijkstra had an unique style of writing, that I always enjoyed reading. His outstanding work helps us understand the evolution of computer science, up to some extent. The University of Texas at Austin (UTA) is kind enough to archive majority of Dr. Dijkstra's writings, and make them available in PDF format, at: In Pursuit of Simplicity: The Manuscripts of Edsger W. Dijkstra web page.

Solaris 10 Update 1 (aka Solaris 10 1/06) is now available for download, at Sun downloads web site. One of the notable features of this major release is the large page support for instructions. It is a known fact that dTLB/dTSB, iTLB/iTSB miss rates is a bottleneck for very large applications. The system has to spend some time serving the virtual-to-physical memory translations. These translations are expensive, and the time spent in these translations are counted in usr% (CPU).

Until the release of Solaris 9, 8K is the only supported page size on Solaris. Solaris 9 introduced Multiple Page Size Support (MPSS), for the applications to use any of the possible hardware address translation sizes supported by the system (use pagesize -a to get the supported page sizes). However on Solaris 9, MPSS for user applications is provided only for anonymous memory eg., heap, stack, ISM, MAP_ANON mappings.

Solaris 10 introduced dynamic TSB support, which helps reducing the dTSB miss rate considerably. However, on Solaris 10, user iTLB misses are still a major bottleneck. In Solaris, user text segments are implemented with memory mapped files, and can potentially be mapped with large pages.

VMPSS - Large Pages for Executables and Files

Now with Solaris 10 Update 1, Sun introduced MPSS for executables (text/instructions) and files, which is referred as MPSS for Vnodes or simply VMPSS. VMPSS extended MPSS support to regular file mappings in order to reduce iTLB misses for text, dTLB misses for initialized data, and also to reduce dTLB misses in general for other memory mapped files. Due to VMPSS, large applications may get large pages automatically, for text and initialized data segments, based on the segment size and the page size support capabilities of the underlying hardware. So if you have a large application running on Solaris 10 Update 1, the application might already be enjoying the benefits of VMPSS, and hence you may observe noticeable improvement in the run-time performance of the application. To check if the application is taking advantage of VMPSS, run pmap -sx <pid>, and observe the page sizes under Pgsz column of pmap output.

Default page sizes, and tunables on different platforms

UltraSPARC II

No large pages are used by default on US-II systems. To enable the default use of 64K or 4M pages for text, add the following lines in /etc/system {and reboot the machine once}:

set use_text_pgsz64k 1 set use_text_pgsz4m 1

To enable the default use of 64K pages for data, add set use_initdata_pgsz64k 1 as well, to /etc/system

UltraSPARC III/III+/IV/IV+

4M is the only large page size supported by default for text mappings. If the application exhibits performance regression with this default behavior, the default use of 4M text pages can be disabled by adding the following line in /etc/system:

set use_text_pgsz4m 0

No large pages are used by default for initialized data segments, on these platforms.

UltraSPARC T1 (aka Niagara)

64K and 4M page sizes for text, and 64K page sizes for initialized data are used by default.

In case of regressions, add the following lines in /etc/system.

To disable the defalt use of 64K or 4M text pages:

set use_text_pgsz64k 0 set use_text_pgsz4m 0

To disable the default use of 64K data pages: set use_initdata_pgsz64k 0

No large pages are used by default on x86. The default use of 2M text pages on PAE machines or 4M pages on non-PAE machines can be enabled with the following setting in /etc/system:

set use_text_largepages 1

----All the above mentioned {VMPSS} design policies should result in the use of large pages when they are most expected to help performance, and limit potential regressions from large pages when they are unlikely to improve performance.

(Note: the words "feedbin" and "feedback" are used interchangeably throughout this blog post)

It is a known fact that by default compiler arranges to create one profile feedback file for each profiled executable. A profiled binary contains compiler inserted instrumentation code, to collect run-time data (like block execution counts) from it. Compiler inserts instrumentation code, when the original code is compiled with -xprofile=collect compiler option.

The default behavior is good enough for small programs or applications with very few executables. However for very large real world applications with tens of executables, having too many feedback files poses slight inconvenience in use phase, where these feedback files are specified on compile line with -xprofile=use:<path_to_feedback_file> option, to generate highly optimized binaries. For example, if the application consists of twenty executables, we need to have twenty -xprofile=use options on the compile line, like:

If make file grabs all compiler flags from environment variables like CFLAGS, it may not be possible to specify all -xprofile=use options in a single CFLAGS, due to the underlying shell restrictions on the number of characters per variable.

The compile line may become too long, and look ugly with too many -xprofile=use flags.

To get around these inconveniences, it is suggested to use compiler supported environment variables SUN_PROFDATA_DIR and SUN_PROFDATA in profile data collection phase, to request the profiler to write all the profile feedback data from different executables in to a single feedbin file, instead of creating one for each executable. If these environment variables are set, the profiler writes the feedback data in to the file pointed by SUN_PROFDATA, under directory SUN_PROFDATA_DIR ie., the feedback data from all executables will be written in to SUN_PROFDATA_DIR/SUN_PROFDATA.

The following example illustrates the default behavior, as well as the behavior with SUN_PROFDATA and SUN_PROFDATA_DIR environment variables.

These variables inform the profiler to write all feedback data from different profiled binaries into a single feedbin file: /tmp/consolidate/singlefeedbin.profile.

%mkdir /tmp/consolidate

%./aIn a.c

%./bIn b.c

% ls /tmp/consolidatesinglefeedbin.profile/

Observe that the singlefeedbin.profile's feedbin holds the feedback data for both executables "a" and "b". If there are more profiled executables, shared libraries, all the feedback data will be appended to this feedbin file.

Here's another instance to show that Solaris is gaining quite a momentum lately, on x86 and SPARC platforms. All these days we were fiddling with RealPlayer binaries targeted for another platform, just to make them work on Solaris x86 platform. Real Player native binaries are now available for both x86 and SPARC versions of Solaris, at Helix community web site. Hooray!

At optimization level 2 (ie., -O2 or -xO2) or higher, Sun Studio C/C++ compilers recognize loop-invariant code and re-arranges it in such a way that it is executed less often (mostly it may get executed only once) than it would be within the loop.

Loop-invariant code consists of statements which could be moved to before or after the loop, without affecting the semantics of the program. In simple words, loop-invariant code is the set of instructions inside a loop whose results do not change; and hence can safely be moved out of the loop.

At O2 optimization level (or higher), compiler recognizes this kind of code, and may rearrange it as follows (check the highlighted code -- if you can understand assembly code, check the assembly by compiling it with cc -S loopinvariant.c):

Note:When -xprofile=use option is used, Sun Studio C/C++ compilers make sure that the loop-invariant code will not be moved from low to high execution places.___________________Technorati tags: C | C++ | Sun Studio | Programming