By Karen Freifeld NEW YORK (Reuters) - A New York state judge on Friday approved most of Bank of America Corp's $8.5 billion settlement with investors over toxic mortgage securities, but left a caveat that could complicate the bank's efforts to implement the deal. Justice Barbara Kapnick ruled that Bank of New York Mellon , the trustee representing investors, had acted mostly in good faith in agreeing to the settlement. But she withheld her approval for one part of the settlement where she said the trustee had not acted reasonably. A spokesman for Bank of America said the bank did not expect that Kapnick's exclusion would hold up the accord. But a lawyer for American International Group Inc , which led investors that opposed the settlement, said the insurer foresees a long legal fight ahead. Bank of America agreed to the settlement in June 2011 to resolve the claims of investors who had bought $174 billion of mortgage-backed securities issued by Countrywide before the U.S. housing crisis. The investors said Countrywide misrepresented the quality of the underlying home mortgages, which went sour in the crisis. Countrywide, based in Calabasas, California, was the biggest home mortgage lender in the United States until the housing market collapsed, specializing in so-called subprime loans, most of which it packaged into securities and resold to investors. It was bought by Bank of America in 2008. A group of 22 investors supported the settlement, including institutions such as BlackRock Inc , MetLife Inc and Allianz SE's Pacific Investment Management Co. But investors led by AIG objected, arguing that they were cut out of negotiations and that there was no evidence the settlement was big enough. In her ruling on Friday, Kapnick wrote that, at the time the settlement was reached, it was clear Bank of New York Mellon was concerned that Countrywide would not be able to pay a future judgment that approached $8.5 billion, and believed it was reasonable to lock in a one-time payment. This was especially so, given that it was "uncertain, at best" whether Bank of America would be held responsible for Countrywide's liabilities, she wrote. Kapnick found that "the trustee did not abuse its discretion in entering into the settlement agreement and did not act in bad faith or outside the bounds of reasonable judgment." Kapnick made one exception in her ruling, withholding her approval from settlement of claims relating to certain loans that Countrywide had modified. Bank of New York Mellon should not have settled those claims without investigating their potential worth, she said. It was not clear what impact the judge's exclusion would have on the settlement. "The issue has the prospect of adding another wrinkle in the timing of the next steps and eventually in determining when the cash flows get paid to bondholders," analysts at Barclays said in a report for clients. The loan modification question was raised by Triaxx funds, which argued during the lengthy proceedings that Bank of New York Mellon failed to investigate claims by investors relating to the modified mortgages. "We're still assessing the impact of the carve-out," John Moon, a lawyer representing the Triaxx entities, said of Kapnick's exclusion. Bank of America shares closed down 1.1 percent to $16.75 Friday. Representatives of Bank of America and Bank of New York Mellon both said they were pleased with the decision. "We believe any outstanding issues raised in the opinion can be addressed without undue delay," said Lawrence Grayson, a spokesman for Bank of America. A lawyer for AIG, Mark Zauderer, took the opposite view. He said the insurer was pleased by the judge's exception for modified loans but disagreed with the other aspects of the ruling. "This case is very far from over because the settlement will not take effect until many potential post-trial motions and appeals are resolved," Zauderer said in a statement. He also called the loan modification issue "critical." Kathy Patrick, who represented the institutional investors who supported the deal, did not return a call for comment The settlement is part of Bank of America's efforts to put liabilities for the financial crisis behind it. It has agreed to pay more than $45 billion to end disputes that came from the financial crisis, including the $8.5 billion Countrywide settlement. Bank of New York Mellon sought judicial approval of the settlement two-and-a-half years ago. As the case dragged on, the number of opponents dwindled. Among them, the attorneys general of New York and Delaware, who intervened in the proceeding in 2011, said last May they would no longer block the accord. Kapnick oversaw a months-long proceeding to determine whether to approve the settlement, which ended on November 21. [ID:nL2N0J5203] Her ruling on Friday came just days before she was to take up a new post on a New York state appeals court on February 3. [ID:nL2N0KR1YX] The case is In re Bank of New York Mellon, New York State Supreme Court, New York County, No. 651786/2011. (Reporting by Karen Freifeld; Editing by Eddie Evans and Ken Wills)