The PROGMEM_PAGE_SIZE define is missing from the iox64a3u.h file. A very
similar MCU, atxmega64a3, has all of the same defines with the addition of the
PROGMEM_PAGE_SIZE at basically the same logical place in its own header file.
My bootloader fails to compile because this define is missing, so I know it
should be there because none of the other Xmegas that I've tried have failed at
this point.
From iox64a3.h:
/* ========== Constants ========== */
#define PROGMEM_START (0x0000)
#define PROGMEM_SIZE (69632)
#define PROGMEM_PAGE_SIZE (256)
#define PROGMEM_END (PROGMEM_START + PROGMEM_SIZE - 1)
From iox64a3u.h:
/* ========== Constants ========== */
#define PROGMEM_START (0x0000)
#define PROGMEM_SIZE (69632)
#define PROGMEM_END (PROGMEM_START + PROGMEM_SIZE - 1)
Adding this to my .c file fixes the issue and leads to successful compilation:
#define PROGMEM_PAGE_SIZE (256)

One additional piece of information related to this:
In the files that have the PROGMEM_PAGE_SIZE define included, the SPM_PAGESIZE
define uses it as follows:
#define SPM_PAGESIZE PROGMEM_PAGE_SIZE
In the header files missing the PROGMEM_PAGE_SIZE define, the value of the
SPM_PAGESIZE define is set directly:
#define SPM_PAGESIZE 256
I'm not sure why this would be the intended behavior, so I'm guessing this was
an oversight or mistake.