This is a "feature" of Solaris' STREAMS-based TCP/IP stack. Basically you have to add a rule allowing the TCP Reset to leave. So let's say that you want to return-rst on ident, port 113, so that sending mail doesn't give long delays, and IRC works:
#return-rst for ident
block return-rst in quick on ppp0 proto tcp from any to any port = 113
pass out quick on ppp0 proto tcp from any port = 113 to any flags R/RSFUP
The first rule just blocks with return-rst, the second rule allows packets out from port 113 with the RESET flag.

Sun has a compiler usually installed in /usr/ucb/cc. Unfortunately, it's a really expensive compiler that doesn't ship with Solaris by default. Fortunately, since Solaris 8, they've included gcc for you on the Solaris Companion CD (/opt/sfw), it's a package called SFWgcc. In Solaris 9 this is now on the main installation CDs and is in /usr/sfw.

If you're using an older version of Solaris you can get gcc from SunFreeware.com.

Note that you need to comment out the "For SUNWSpro" lines in the Makefile and uncomment the "for GCC" lines in order to compile with gcc.

You're most likely trying to use an IPF modules compiled as a 32-bit binary. You need to compile it as a 64-bit binary, so you must either use the cc with SUNWspro, or GCC 3.0 (see VII-5). GCC 2.x will NOT work. SUNWspro is available from Sun, you can get a demo license (please READ what you can and cannot do according to the demo license).

Sun's Forte Compiler can make 64 bit modules. This compiler is not standard with the OS, you have to buy it separately. However, there is a "try-and-buy" version which you can install, and use for a limited time. You can get this time limited version on cdrom or you can download it from Sun.com. Note that the download is very large. Please READ what you can and cannot do according to the demo license.

Solaris's /etc/system is consulted when booting, so you can modify IPF kernel parameters there, for example:

*
* ipf: adjust the default tcp timeouts downward so that
* idle (dead) and half closed states get killed off quicker.
set ipf:fr_tcpidletimeout = 172800
set ipf:fr_tcphalfclosed = 7200
*
* ipf: adjust the state table sizes so we have enough buckets.
* IPSTATE_MAX (=fr_statemax) should be ~70% of IPSTATE_SIZE
* IPSTATE_SIZE (=fr_statesize) has to be a prime number
set ipf:fr_statemax = 7000
set ipf:fr_statesize = 10009
*
* ipf: adjust the NAT table sizes so we have enough buckets.
* generally you have fewer than 127 rules in ipnat.conf
* so no need to waste memory for more.
set ipf:ipf_nattable_sz = 10009
set ipf:ipf_natrules_sz = 127
set ipf:ipf_rdrrules_sz = 127
*
* note that the timers run "2 ticks to a second", so
* for example, written below is the following:
* set ipf:fr_tcpidletimeout = 172800
* this sets the tcp idle connection timeout to
* (172800/2) / 3600 = 24 hours.
*

There are three possiblities here: 1) using Solaris curses, 2) using ncurses from the Solaris Companion CD 3) using ncurses you compile yourself. For all three of these methods make sure the following is set (3.4.22 and on already has this):
STATETOP_CFLAGS=-DSTATETOP

Once that's done, pick one of the three methods above. The easiest way is to use Solaris curses. For that method, in the Makefile change:
STATETOP_LIB=

to say:
STATETOP_LIB=-lcurses

And that will do it.

If you don't want to use Solaris curses and would prefer to use ncurses, install the ncurses package on the Solaris 8 Companion CD or from any other source of your choice and then change your STATETOP_INC line to read:
STATETOP_INC=-L/opt/sfw/include
and set:
STATETOP_LIB=-L/opt/sfw/lib -R/opt/sfw/lib -lncurses

Adjust accordingly if your ncurses libs/includes are in a different place.

If you want to use ncurses but are using Solaris < 8, or just don't want to use the Companion CD verion of ncurses for some reason, then adjust the STATETOP_INC line to read:
STATETOP_INC=-I/usr/local/include

And set STATETOP_LIB to be:
STATETOP_LIB=-L/usr/local/lib -R/usr/local/lib -lncurses

Note that these are usual locations for the include and library files. If you installed them in other places, you'll need to specify the appropriate path's.

NOTE: If you are using gcc 3.1+, you may need to uninstall either curses, or ncurses. Having both may cause conflicts during compile.

According to SunSolve many of these tunnels use a ton of space in the stack, so you should increase your stack size. To find out what it is do:
echo 'lwp_default_stksize/D' | adb -k /dev/ksyms /dev/mem

Solaris 2.6 and above in 32-bit mode default to 0x2000 while Soalris 7 and above in 64-bit mode default to 0x4000. Try doubling this number. To set it, add a line to /etc/system like this:
set lwp_default_stksize=0x4000

and reboot. You may also use decimal values (0x4000 = 16384 and 0x8000 = 32768). For more information on tuning kernel parameters click here, and for more information on lwp_default_stksize click here.

There are no official IP Filter binaries. However some kind people have made their binaries available for download. NOTE WELL: The following sites are NOT OFFICIAL. The binaries there are NOT supported by Darren Reed, Phil Dibowitz, OR the authors or owners of the sites (unless they state otherwise).

From Darren Reed: "If you're using IPFilter on Solaris9, you might want to make sure you apply patch 112233-02 (or later) to fix a problem with the kernel attempting to prevent too much stack being used (and causing the system to crash.) This is particularly fatal when using IPFilter with ip.tun* and ESP+AH."

The problem is that the Solaris headers changed across updates of Solaris 9 and you are using a GCC from before the change on an updated system. (i.e. a GCC built for Solaris 9 <= 12/03 on Solaris 9 >= 4/04).

From Darren:
"Very significant. I did do some benchmarking of this, originally but I forget what the performance measurements were, now.

"The improvements in performance come from two areas. The first is that the packet matching is now all in C, rather than using intermediate structures. The second is that rather than compare each field, one at a time, in each rule, it sorts the fields to be matched for each rule as an optimisation and only does comparisons when the matching is different."