In a recent lecture, Shaykh Dr Yasir Qadhi mentioned (slightly paraphrasing) "Our scholars of the past said, if these rich people committing these sins had known the sweetness and the piece we feel, they would be fighting with us, cutting our chests to get that sweetness, but it does not work that way." Would anybody know a concrete source of this?

One day he said to some of his companions: if the kings and the princes had known -meaning the happiness and delight- what we feel, they would have whipped us by swords

I found this statement also attributed to al-Hassan al-Basri and to people of wisdom in general (Ibn Qayyim quoted it several times in his books without addressing it to a special person).

This was commented by Ibn Taymiyyah (as quoted by ibn Qayyim al-Jawziya in madarij as-Salikeen) in a similar wording as your own statement:

لو يعلم أصحاب الدنيا والمال والملك والجاه والسلطان لقاتلونا عليها، لأن السعادة في نظرهم هي التمتع بملاذ الدنيا من أكل وشرب ونساء، وهذه هي الغاية التي يريدونها من السعادة.
If the people of dunya (this life) and rich people, kings, people of eminence and authority knew about what we feel they would fight with us to get it. because happiness in their opinion is the pleasure from the wealth of dunya, like eating, drinking, women and this is the goal they look for when seeking happiness.

In his sayd al-Khatir ibn al-Jawzi commented the statement of Ibrahim ibn Ahdam saying (My own translation take it carefully):

" ولقد صَدَقَ ابن أدهم، فإن السلطان إن أكل شيئاً خاف أن يكون قد طُرِحَ له فيه سم، وإن نام خاف أن يغتال، وهو وراء المغاليق لا يمكنه أن يخرج لفرجة، فإن خرج كان منزعجاً من أقرب الخلق إليه، واللذة التي ينالها تبرد عنده، ولا تبقي له لذة مطعم ولا منكح.
Ibn Adhma was correct, as the sultan if he wanted to eat something was frightened that an enemy might have put some poison in it, if he wanted to sleep he was frightened to be killed or murdered, and he is behind the walls (like a prison) and can't leave them to look at spectacle, and if he left he might be displeased by the closest people to him, and any (hot) pleasure he gets would cool down so he can't find pleasure in neither food nor in women (legal intercourse).

وكلما استظرف المطاعم أكثر منها ففسدت معدته، وكلما استجد الجواري أكثر منهن فذهبت قوته، ولا يكاد يبعد ما بين الوطء والوطء فلا يجد في الوطء كبير لذة لأن لذة الوطء بقدر بعد ما بين الزمانين، وكذلك لذة الأكل.
And whenever he enjoys food he would eat too much and harm his stomach, and whenever he gets new female slaves he would want to take too much so that his strength would leave him and he wouldn't make a pause between one intercourse and another to the extent he won't find any pleasure in it and the same would apply to the pleasure of food.

فإن من أكل على شبع ووطىء من غير صدق شهوة وقلق لم يجد اللذة التامة التي يجدها الفقير إذا جاع والعزب إذا وجد امرأة.
As those who eat after fullness or have intercourse without real lust wouldn't get the full pleasure that a poor individual who is hungry or unmarried would feel if he encountered a woman (that might be suitable).

Ibrahim ibn Adham my Allah have mercy with him has said: We by Allah are the rich kings , we are thoise who didn't hasten for peace of mind in dunya, and didn't care about which state we woke up or went to bed at night as long as we have obeyed Allah may He be exalted and glorifled!

The closest sourced statement that looks like this can be found here in ibn 'Asakir's tarikh Dimashq were it is quoted as a stamen of Ibrahim ibn Adham.

Actually the statement in al-Hilya (See here) is much longer and quoted as a narration (with a disconnection in the chain) on the authority of Ibrahim ibn Bashar al-Khorasani the servant of ibn Adham: