Any css transitions that run depend on the element being visible with display:block or similar.

By calling
document.getElementById("modal").style.display="none"; at the start of your close() function you instantly set the entire modal to be completely hidden, so the transition on the modalcontent has no effect.

Likewise in the open() function, the transition class is applied but the element is hidden by display:none so the transition does not run.

You should try to have the transition run, then after a delay set the modal to be hidden.