Some of those files are used to restore the backup or to make incremental backups for example.

xtrabackup_checkpoints will store the LSN range of this backups. It is used for incremental backups..
xtrabackup_binlog_info will contain the the bin log file and position when you made the backup. It will be useful if you want to create slaves from this backup
xtrabackup_binlog_pos_innodb if the same as xtrabackup_binlog_info but only valid if you only have XtraDB of InnoDB tables.
xtrabackup_binary xtrabackup used to make the backup.
xtrabackup_logfile used to restore your backup.

I don't understand why you might want to remove those files from a backup.. but not.. this is not recommended.