Example: If I as a hunter invest in the BM tree, will it only buff my pet for as much as the talents state? or is there some sort of a hidden increase to my pet's damage / survivability proportional to the amout of talents invested in that tree? is there such game mechanic in action in classic wow talent system?

I'm not quite sure I understand the question. I guess the "hidden" talents would be the base value of your pet and its own abilities which can be different depending on what pet you tame? Another consideration would be the rank of each of your pets abilities which can drastically affect his throughput.These are hardly hidden though and can be found on petopia (ex. Brokentooth's fast attack speed - likely will be present in Classic / Lupos doing shadow dmg - likely wont be present in Classic) or found ingame by using your beast lore ability.

Otherwise I dont think there is anything "hidden" in the talent tree itself. "is there some sort of a hidden increase to my pet's damage"... The only other "hidden" modifier I can think about your pets damage would be spell effects that can be placed on the pet like Juju's and scrolls. This is more of a min/max thing than it being hidden, often avoided by hunters due to the shear cost/effort of doing it consistently.

The amount of talents you have in Beast Mastery does not increase any pet skills passively or hidden. Your pet will have the same stats if you have 30 points in BM or 0. Same with your hunters damage via bows or guns with talents in the marksman tree.