The formulas for the distortion parameters are given in the Appendix C of PhotoScan Manual, and there's no straightforward conversion between distortion models used in different packages.

If you need them in PhotoModeler format, then you need to save calibration in XML format in Agisoft Lens and use PhotoScan to convert it to the desired format via Tools Menu -> Camera Calibration window.