In 10.5 Mac OS X added enough utmpx support to make it past the first two preprocessor checks around line 390 of src/smbd/utmp.c and on to the utmpx block which uses updwtmpx(). Unfortunately, as ./configure correctly surmised, 10.5 doesn't have that. Seems to be a straightforward logic error in the preprocessor guard checks, fixable with the patch below. (Not sure if this is Odd or not in the comment.)
--- smbd/utmp.c.orig 2008-01-09 20:25:53.000000000 -0500
+++ smbd/utmp.c 2008-01-09 20:30:08.000000000 -0500
@@ -398,6 +398,10 @@
/* Odd. Have utmpx.h but no "getutmpx()". Drop to non-x stuff */
DEBUG(1,("utmp_update: have utmpx.h but no getutmpx() function\n"));
utmp_nox_update(u, claim);
+#elif !defined(HAVE_UPDWTMPX)
+ /* Have utmpx.h but no "updwtmpx()". Drop to non-x stuff */
+ DEBUG(1,("utmp_update: have utmpx.h but no updwtmpx() function\n"));
+ utmp_nox_update(u, claim);
#else
pstring uname, wname;
struct utmpx ux, *uxrc;