I'm writing a bash script that adds PPAs automatically. The trouble is if you just run the add-apt-repository ppa command then you will get a bunch of duplicate commented lines in each PPA list file in the /etc/apt/sources.list.d folder after running the same script a few times.

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

That question is from early 2013, are you sure add-apt-repository is still stupid enough to duplicate source list entries? I am pretty sure (but can't check at the moment) that it aborts with a message if the source is already listed.
– Byte CommanderAug 6 '16 at 18:43

@ByteCommander Sadly yes it is that stupid. I'll add an example at the bottom of the question. Lol, maybe I should go find the source for apt-add-repository and try to get a pull request merged in.
– Simon The CatAug 6 '16 at 18:47

@ByteCommander Not in the real sense, but after I run my script a bunch of times its gonna use up a lot of space needlessly. Why keep more then one of the same thing commented out?
– Simon The CatAug 6 '16 at 18:53

A few bytes are not really a lot of space nowadays... But I agree.
– Byte CommanderAug 6 '16 at 18:56

1 Answer
1

Please don't do this. See the link in the section title for an explanation why and possible alternatives. The easiest and most elegant solution here would be to use shell globbing to generate the items to iterate over in a for loop:

Transforming files in-place

We all know that reading a file with one program and redirecting the output back into that same file likely results in an empty file. Throwing tee in between helps a little because it buffers input for a while until it writes it to its output, but that's an implementation detail that may fail with different tee implementations and/or different kernels. It also fails as soon as the size of the output exceeds the size of the buffer. Instead you should use something like

uniq "$l" | sudo sponge "$l"

or possibly

sudo sed -i ... "$l"

if you can find an appropriate sed script to do the job.

Both have the added benefit that they don't also write the input to the standard output, but you can suppress that with >/dev/null anyway.

Does uniq really find duplicate Apt source lines?

Looking at sources.list(5), Apt source lines can contain multiple “components”. The following sources files are equivalent but not identical:

It should be obvious that therefore uniq cannot find all duplicate Apt source lines. However, since repositories on ppa.launchpad.net don't support multiple components, you may get along if you only use PPAs from that site. Additionally, duplicates may be spread over multiple files, which breaks this approach entirely unless you rely on the fact that add-apt-repository manages source files somewhat intelligently and puts exactly one PPA in each file.