He might see his yard as part of his living space that he doesn't want to "dirty" up.

I'd just stop taking him on so many walks, and give him that time in the yard instead. You'll have to keep a close eye on him the first few days to make sure that he doesn't have an accident in the house, but he will have to go eventually and when he does you'll want him to be in the yard.