A botnet is a collection of compromised computers, each of which is
known as a 'bot', connected to the Internet. When a computer is
compromised by an attacker, there is often code within the malware
that commands it to become part of a botnet. The "botmaster" or "bot
herder" controls these compromised computers via standards-based
network protocols such as IRC and http.

Is there a way to detect that your computer is being used in a botnet-based DDoS attack?

There is some tool such as a software that could detect strange traffic and exploit activity from my computer?

Analyzing kernels and executables before they are run is usually a safe practice, but it is best combined with a strong OS access control model, such as SELinux.

While it is difficult to locate backdoors in self-modifying or self-checking code, usually the presence of self-modifying or self-checking code themselves are an indicator of malware. For example, the MANDIANT Red Curtain tool can aid a malware analyst in detecting high entropy in sections of code, or strange names/patterns in these sections.

Additionally, many other practices exist (and not all are documented), such as tracing, emulation, et al. Combining all of these methods into one tool or technique is not common, as malware analysts often shift from static to dynamic analysis and back very quickly (in order to gain perspective). However, a staged approach using a minimal set of tools can accomplish a lot in a very short amount of time. For example, I often use pescanner.py on a PE executable before running it in a sandbox along with PE process memory dumpers and instrumentation for tracing such as PIN.

Some of the recent literature has been increasingly good in this area of research.

The only way to detect it would be to sniff the traffic between the machine and the internet, using an external, known-good machine. Even then there would be no way to know if the bot was just dormant at the time you looked for it.

Another option is to boot the machine from known-good (preferably read-only) media (e.g. CD or DVD) and do forensic analysis of the machine to look for traces of an infection.

First, you can't trust a compromised computer as the output of all utilities could be faked to hide the presence of malicious activities.

I would generally not expect any computer with robust security practices to be part of a bot net. One where you only install code from trusted sources, have strong passwords (that aren't used elsewhere), etc. Unless my ISP says they've received reports of unusual activity from my IP address indicating a potential bot, or a user tells me they were tricked into installing software that didn't do as advertised and they now realize was likely a trojan.

In those cases, I would start testing for signs of a potential bot:

monitor network activity. Run netstat -- do all connections more or less make sense? (ignore local connections; run whois on foreign IPs I don't recognize). Remember netstat's output could be faked, so you also run wireshark (network analyzer) from another computer (presumed safe) on my local network, and check to see if there's any unusual traffic to/from strange non-local IP addresses (that's not routine stuff from my router/network printer, etc)?

Boot into a live cd of the same distribution (so you trust the utilities), and run checksums (md5) on important utilities (/bin, /sbin/, /usr/bin/, etc.) and compare to known checksums of a safe computer with the same OS / version (or make a fresh install and run all updates and then generate checksums to compare with). (If the checksums don't match, it could possibly be due to an update of the utility; so check versions and investigate further). I've assumed linux/unix system for this answer, but presumably you could do the same for windows if you know what to check.

If you find evidence of a bot, I'd back up my data (that is non-executables), wipe the hard disk clean, and re-install the OS from a safe source with different passwords. I'd be very wary opening up documents that could contain macros (e.g., doc, pdfs) from the compromised computer (making sure to disable macros in the relevant programs before looking at these files again, not being logged in as administrator/root, and installing a virus scanner).

A darknet is also a good way to help discover infected machines on your network. It is essentially a subnet on your LAN that shouldn't normally have traffic routed to it, with logging machines in it, which detect which machines aren't obeying your normal network setup, e.g. scanning for nodes on the network to infect.

Since you should know what behaviour is normal and what isn't, infected machines trying to infect others will stick out like a sore thumb.