Is it ever rational for a country to undertake to abide by "rules of war", such as the Geneva Conventions?
One could argue certain advantages to be gained as long as a country can expect reciprocity, e.g.,

decent treatment for one's prisoners in exchange for like treatment of enemy prisoners

avoidance of the use of certain weapons knowing that the consequences could be devastating if the enemy responded in kind

By their nature, such rules are only enforceable by self-policing (except by the victors after the war), yet in certain theaters (e.g., the Western Front in World War II), they were to a large degree observed.