By Karen Freifeld

NEW YORK, Jan 31 (Reuters) - A New York state judge on Friday approved most of Bank of America Corp's $8.5 billion settlement with investors over toxic mortgage securities, but left a caveat that could complicate the bank's efforts to implement the deal.

Justice Barbara Kapnick ruled that Bank of New York Mellon , the trustee representing investors, had acted mostly in good faith in agreeing to the settlement. But she withheld her approval for one part of the settlement where she said the trustee had not acted reasonably.

A spokesman for Bank of America said the bank did not expect that Kapnick's exclusion would hold up the accord.

But a lawyer for American International Group Inc, which led investors that opposed the settlement, said the insurer foresees a long legal fight ahead.

Bank of America agreed to the settlement in June 2011 to resolve the claims of investors who had bought $174 billion of mortgage-backed securities issued by Countrywide before the U.S. housing crisis. The investors said Countrywide misrepresented the quality of the underlying home mortgages, which went sour in the crisis.

Countrywide, based in Calabasas, California, was the biggest home mortgage lender in the United States until the housing market collapsed, specializing in so-called subprime loans, most of which it packaged into securities and resold to investors. It was bought by Bank of America in 2008.

A group of 22 investors supported the settlement, including institutions such as BlackRock Inc, MetLife Inc and Allianz SE's Pacific Investment Management Co.

But investors led by AIG objected, arguing that they were cut out of negotiations and that there was no evidence the settlement was big enough.

In her ruling on Friday, Kapnick wrote that, at the time the settlement was reached, it was clear Bank of New York Mellon was concerned that Countrywide would not be able to pay a future judgment that approached $8.5 billion, and believed it was reasonable to lock in a one-time payment.

This was especially so, given that it was "uncertain, at best" whether Bank of America would be held responsible for Countrywide's liabilities, she wrote.

Kapnick found that "the trustee did not abuse its discretion in entering into the settlement agreement and did not act in bad faith or outside the bounds of reasonable judgment."

Kapnick made one exception in her ruling, withholding her approval from settlement of claims relating to certain loans that Countrywide had modified. Bank of New York Mellon should not have settled those claims without investigating their potential worth, she said.