For this encounter, it is likely the devourer will have plenty of time to prepare and plan, which means it can use lesser planar ally to summon some additional support. The question is, how much support?

Note that the spell requires the caster to pay the called creature, and next note that the devourer itself is listed as having no treasure. Technically, the devourer can't use the spell unless you assume it has some treasure or can pay the called creature in some other way. The problem is that at a bare minimum, the devourer could theoretically use the spell thirty times, gather up 30 outsiders of 6HD each as long as it can make them all payments, and then gang rush a party of PCs.

So what should be a reasonable cap on the number of outsiders a devourer can use? I am thinking somewhere between 2 and 5, but I am guessing.

What's reasonable? Whatever keeps the Encounter Level appropriate for the party. So what level is the party and how many are in it?

Also note that summoning spells are typically given to monsters as a way to call allies during combat. Otherwise, it's a bit pointless to even consider it a summoning spell. If it happens off-screen, it's equivalent to the Devourer just hanging out with those monsters. If you want it to have buddies, just give it buddies. If you want the allies to show up in the middle of the battle and surprise the party, have him use the SLA during combat. Maybe give him a couple spots on the battlefield where he's relatively safe and can take the standard action to use it.

Agreed. If he summons the creatures outside of combat, they should count toward the encounter level the same as if they were normal creatures. I once ran a battle against a psion shaper and a bunch of Astral Constructs, but they were permanent constructs; the fact that he ostensibly manifested them was pure flavor, as they would have lasted for like a minute and a half each if they'd been using the rules, and thus he couldn't very well have had them all up at the start of the battle.

Thanks guys for showing me the key point. The creatures are only counted as part of the CR 11 devourer if he actually summons/calls them during combat where he uses up his action in a round to do so. If he instead brings them into play before the PCs even encounter him, then it is the CR 11 devourer plus the CR of each of his summoned allies. In this case, I plan to have him summon canoloths...the fan-made revision here...

So assuming four standard 11th level characters, the devourer itself should challenging. Throwing in additional CR 6 canoloths will quickly move up the encounter to very difficult. I will still keep it to five canoloths or less.

I would also stress that 3.5's Encounter Level and, really, Challenge Rating system... isn't particularly accurate once you start pulling from books outside of the Player's Handbook and Monster Manual. And even for the core books, summoning always was a headache. Even if it spends an action in combat to summon a creature... that creature then gets a turn. I would hesitate to say that a Devourer that summons 1 creature in battle has the same Encounter Level as a Devourer that summons 6 creatures in battle, even if it uses its action to summon it. I know 3.5 says the Encounter Level doesn't change. I know it says that depletion of the summoner's resources and actions is accounted for. But frankly, I think that's somewhat crazy.

A CR 11 Devourer with five of those CR 6 homebrew Canoloths is going to be nearly impossible for a four-person CR 11 party. If that's what you're going for... then okay. But there's a fair chance that it ends in a TPK ridiculously quickly. If nothing else, they out outnumber the party and have advantage with action economy. But also those Canoloths look a bit more dangerous than CR 6 to me.

Fiends typically pay each other in favors owed, but a devourer has in its possession a currency particularly attractive to creatures of the lower planes: souls. It'd be reasonable to expect a devourer to give a captured soul to each fiend it summons.

Yeah, I ran some checks with the the d20 online encounter calculator and it really doesn't distinguish between 2 and 12 additional CR6 opponents, which is just ridiculous. I use it as a guideline and I know what my party is capable of. I stated four 11th level characters to keep things simple in theory, but I actually have six PCs, four are 11th level and two are 10th level. They are skilled tacticians and play some of the more potent Tier classes in 3.5 and I know the devourer is going to get overwhelmed quick if he doesn't have some support. My goal is a battle that uses the majority of the party's resources for the day forcing them to withdraw and regroup.

The players do have shot at engaging the devourer before it can prepare if they play their cards right, we will see how things go.