“This Court should reject the Proposed Settlement in its current form and encourage the parties to continue negotiations,” reads a filing the DOJ submitted to the U.S. District Court for the Southern District of New York late on Friday.

The proposed settlement must be modified by Google and the plaintiffs so that it complies with U.S. copyright and antitrust laws, the DOJ said in its filing, a 32-page Statement of Interest.

The DOJ also expressed concern that the proposed settlement doesn’t satisfy the requirements of Federal Rule of Civil Procedure 23, which sets parameters for approving settlements of class action lawsuits.

The DOJ’s opinion is likely to be seen a major blow against Google, which has been in court over this matter for about four years and is eager to get the case settled.

“We are considering the points raised by the Department and look forward to addressing them as the court proceedings continue,” Google and the plaintiffs—the Authors Guild and the Association of American Publishers (AAP)—said via e-mail in a brief joint statement reacting to the DOJ’s filing.

The DOJ had until Friday to submit a written report to the court on the findings of a formal investigation over whether the proposed agreement violated U.S. laws.

In fact, the DOJ’s Antitrust Division continues investigating the settlement, the DOJ said in Friday’s filing, which nonetheless contains “a preliminary explanation” of the agency’s antitrust concerns.

Despite its criticisms, the DOJ said it recognizes that a settlement of the case would benefit the public by making millions of hard-to-find books easily accessible in digital form, something Google and the plaintiffs highlighted in their statement.

The DOJ said in the filing that it is “committed to working with the parties constructively with respect to alterations the parties may propose.”

“Because a properly structured settlement agreement in this case offers the potential for important societal benefits, the United States does not want the opportunity or momentum to be lost,” the filing reads.

Rumors that the DOJ had started reviewing the proposed agreement surfaced in April. The DOJ confirmed that it was conducting a formal investigation in July.

Those following the case have been widely anticipating the result of the DOJ’s probe of the proposed agreement, which has been loudly praised and criticized since it was announced in October of last year.

In 2005, book authors and the Authors Guild filed a class action lawsuit, while five large publishers filed a separate lawsuit as representatives of the Association of American Publishers’ membership.

The lawsuits were brought after Google launched a program to scan and index sometimes entire collections from the libraries of major universities without always getting permission from the copyright owners of the books.

Google made the text of the books searchable on its book search engine, claiming it’s protected by the fair use principle because it only showed snippets of text for in-copyright books it had scanned without permission.

However, after two years of negotiations, Google and the plaintiffs reached middle ground, hammering out a wide-ranging settlement agreement that calls for Google to pay $125 million and in exchange gives the search company rights to display meatier chunks of these in-copyright books, not just snippets.

In addition, Google would make it possible for people to buy online access to these books. The agreement would also allow institutions to buy subscriptions to books and make them available to their constituents.

A royalty system would also be set up to compensate authors and publishers for access to their works via the creation of the Book Rights Registry. This would be an independent, nonprofit entity entrusted with distributing payments to copyright holders earned through online access to their works. Revenue will come from institutional subscriptions, book sales and ad-revenue sharing.

The Registry, whose board of directors would be made up of an equal number of author and publisher representatives, would also locate and register copyright owners, who in turn have the option to request to be included in or excluded from the project.

A big portion of Google’s $125 million payment would go towards funding the Registry, while the rest would be used to settle existing claims by authors and publishers and to cover legal fees.

Google, the Authors Guild and the AAP maintain that the proposed settlement will be beneficial to authors, publishers and readers by making it easier to find, distribute and purchase books, especially those that are out of print.

However, critics have raised several objections, including what they perceive as excessive control by Google over prices and over so-called “orphan works.” The latter are books that are under copyright but whose owners can’t be found because the author has died or the publishing house disappeared.

The court where the case is being heard allowed hundreds of backers and critics of the proposed agreement to submit opinions for several months. That comment period closed earlier this month.

In its filing, the DOJ urged the court to tread carefully because the agreement is one of the most “far-reaching” class-action lawsuit settlement proposals the agency is aware of.

“A global disposition of the rights to millions of copyrighted works is typically the kind of policy change implemented through legislation, not through a private judicial settlement. If such a significant — and potentially beneficial — policy change is to be made through the mechanism of a class action settlement — as opposed to legislation — the United States respectfully submits that this Court should undertake a particularly searching analysis to ensure that the requirements of Federal Rule of Civil Procedure 23 — Rule 23 — are met and that the settlement is consistent with copyright law and antitrust law. As presently drafted, the Proposed Settlement does not meet the legal standards this Court must apply,” the filing reads.

According to the DOJ, Rule 23 is designed to ensure that settlements of class action lawsuits resolve disputes for plaintiffs with aligned interests. It also seeks to protect the legal rights of absent class members with divergent interests from the class representatives. The proposed agreement falls short in this regard, according to the DOJ.

“The most sweeping forward-looking licensing provisions of the current Proposed Settlement — which give open-ended control to the Registry and Google for the exploitation of the rights of absent class members unless those class members opt out of those provisions — both exacerbate potential conflicts between the interests of the class representatives and those of absent class members — especially rightsholders of out-of-print works and foreign rightsholders — and are difficult to square with the requirements of Rule 23,” the filing reads.

Since the DOJ hasn’t completed its antitrust investigation, it can’t state that the proposed agreement violates antitrust law, but it does have serious concerns in this area. For starters, it seems to the DOJ that the agreement gives book publishers power to restrict price competition. The DOJ is also worried the agreement could shut out other digital distributors from competing with Google.

Consumer Watchdog, a consumer protection organization that earlier this year urged the DOJ to get involved, filed a 30-page document opposing the agreement, saying it will “strip rights from millions of absent class members, worldwide, in violation of national and international copyright law, for the sole benefit of Google.”

There should be a competitive book-search market, while the U.S. Congress must solve the orphan works problem, according to the group. “The parties simply cannot justify this ‘solution’ which does not adequately protect the Rightsholders and unfairly benefits a single party,” reads the Consumer Watchdog statement.

Meanwhile, the Center for Democracy and Technology (CDT) came out in favor of the deal, saying the new book search services will be “extraordinarily valuable, and will make available to the public a vast amount of knowledge and information that is largely inaccessible today.” The CDT tempered its endorsement by stating that the new services create “serious privacy concerns” and that the court should take “affirmative action” during the settlement process to make sure reader privacy is protected.

On Oct. 7, Judge Denny Chin will preside over a hearing where Google and the plaintiffs will have a chance to offer oral arguments in favor of approving the agreement. Those opposed will also have a chance to voice their opinions.

To comment on this article and other Macworld content, visit our Facebook page or our Twitter feed.