High-kappa cerium oxide (CeO2) was applied to AlGaN/GaN high-electron-mobility transistors (HEMTs) as a gate insulator and a passivation layer by molecular beam deposition (MBD) for high-power applications. From capacitance-voltage (C-V) measurement results, the dielectric constant of the CeO2 film was 25.2. The C-V curves showed clear accumulation and depletion behaviors with a small hysteresis (20 mV). Moreover, the interface trap density (D-it) was calculated to be 5.5 x 10(11) eV(-1)cm(-2) at 150 degrees C. A CeO2 MOS-HEMT was fabricated and demonstrated a low subthreshold swing (SS) of 87 mV/decade, a high ON/OFF drain current ratio (I-ON/I-OFF) of 1.14 X 10(9), and a low gate leakage current density (J(leakage)) of 2.85 X 10(-9)Acm(-2) with an improved dynamic ON-resistance (R-ON), which is about one order of magnitude lower than that of a conventional HEMT. (C) 2016 The Japan Society of Applied Physics