As the commentators point out below, another key difference is for the longer form

“Evaluation proceeds only until the result is determined”

This concept is highlighted in the following example:
> f = function(){cat("My name is f\n");return(TRUE)}
> g = function(){cat("My name is g\n");return(FALSE)}
> f() | g()
My name is f
My name is g
[1] TRUE
> f() || g()
My name is f
[1] TRUE

This has two benefits:

Evaluation will be faster. In the above example, the function g isn’t evaluated (thanks to Andrew Robson and NotMe)

Also, you can use the double variety to check a property of a data structure before carrying on with your analysis, i.e. all(!is.na(x)) && mean(x) > 0 (thanks to Pat Burns for this tip)