Know what authors are good at? Writing. They're not [necessarily] good at layout, or editing, or marketing, or distribution, or cover design, or any of these other things.

As a self-published author who has also been successful with traditional publishing, let me tell you: these things are hard, painful, and expensive.

By taking out the publishers, you're taking out the people who are least somewhat good at these things.

Don't you want to select from good writers, not the much smaller subset of writers who are also good at layout, editing, distribution and marketing?

2. Publishers are a quality signal.

With or without publishers, there will be no shortage of books in the world.

Publishers offer value as a gatekeeper or a curator of not-totally-terrible stuff.

How many self-published books have you read recently? Do you want to have to dig through loads of bad books to find the occasional good one?

3. Authors can already sell to customers directly.

Publishers are optional. You don't have to use one. An author can just go plop their book up on Amazon, taking care of the editing/layout/cover/marketing themselves.

Publishers offer a service to authors. Why is the existence of this service a bad thing for authors or readers?

4. You don't want to give Amazon that much power.

If/when publishers go away, their "power" doesn't really go to authors. It goes to Amazon. A world without publishers means a world in which one enormous company has monopoly power that it can essentially abuse against authors, who have little choice other than to work with Amazon.