Well there are a number of ways that this could be done. If one were to raise a large number of fireflies of those there is a good chance, by chance alone that a few of them, would have a mutation that causes it not to glow. Then of the glowing and non-glowing, the DNA could be compared for mutations.

I’m having trouble with the question(s). Time used to be that genes were identified by phenotype—and sometimes only by phenotype. Nowdays we have literally millions of bases of cDNA and genomic sequences sitting around in databases with no known gene function ascribed to them, though they are sometimes annotated as being similar to other gene sequences with known biochemistry or genetics. I get frustrated trying to come up with a rational answer to your question(s) that isn’t 5 pages long. My first impulse was to ask you to re-phrase the question—actually, my first impulse was to ignore the question, but as not many have replied, I thought I would take a stab at it, though I’m not comfortable I understand exactly what you are asking.

“do they compare genes from two species and then see what ones are different”

Yes, but what do you mean by comparing genes? Looking at the sequence similarities (DNA or amino acid?) between genes or purported genes, I suppose you mean. Unless you are comparing highly conserved genes (and even then you can expect to see changes), most likely there will be differences between any two homologous/orthologous sequences from different sources. The more closely related the sources, the less likely there will be many differences, and conversely, the more evolutionarily distant the sources, the more likely there will be a lot of sequence differences.

Most of the time you won’t have any idea what these sequence differences mean phenotypically. They say that Max Perutz knew and understood the structure and function of hemoglobin so well he could predict the phenotype of arbitrary changes to the amino acid sequence. Rarely do you have so much information as that (and even more rarely will there be someone like Max Perutz around to interpret for you.) So I really don’t know how you would decide if the differences you might see comparing two (or more) sequences might “make a difference,” and in what way do you mean?; obviously if they are not identical, then they are by definition “different,” but what difference that makes is anybody’s guess most of the time. There will most likely be a simple answer to why your hypothetical beetle doesn’t glow (if there even are closely related beetles); it probably lacks a functional luciferase gene (or fails to make the glowable substrate for some reason). But we know enough about luciferase to go looking for luciferase homologues. It’s the other expressed sequences we don’t know very much about that are the problem and why there is a huge proteomics effort springing up alongside the genomics efforts that have been going full-tilt for perhaps a decade now.