I install lots of third party software on my ubuntu system. I wonder if having 50-60 repos added to "software sources" slows down my system in any way? On the other hand, after installing a package, if I remove its source, will it affect anything?

Agree with the comment above, I would be curious as to why you need that many third party repos?

One other minor point to consider is that the third party repos may not be maintained in the same manner as the repos directly from Ubuntu. As stated on Ubuntu.com

Make sure that all repositories you add in this way have been tested and are known to work on Ubuntu systems. Repositories that are not designed to work with your version of Ubuntu can introduce inconsistencies in your system and might force you to re-install.