Media mogul Barry Diller has taken on an ambitious $200 million project to build a park, called Pier 55, along the Hudson River. There’s just one thing standing in his way: He’s being sued by the City Club of New York, a civic group founded in 1892.

“This group of three people sued us,” Diller said in the closing panel of Vanity Fair‘s New Establishment Summit on Thursday.

In the beginning, Diller said, they sued him for odd things, like for impeding light from reaching a special fish in the Hudson. Diller said he was perplexed by these strange lawsuits until he found out who was behind the City Club of New York.

“I couldn’t figure it out until I realized the three people — one of whom was named Durst (Doug Durst), from the very honorable Durst family killing machine,” said Diller, who is chairman and senior executive of both IAC/InterActiveCorp (IACI) and Expedia (EXPE).

Diller was making a reference to Robert Durst, who is suspected of multiple murders and is also a scion of the wealthy real estate family. Durst was the subject of the hit HBO documentary mini-series “The Jinx,” which examined his connection to those murders.

On Thursday, Vanity Fair editor Graydon Carter tried to correct Diller when he described the entire clan as a killing machine. “No, that’s one member — Robert Durst,” Carter said.

“He should have killed his brother,” retorted Diller.

Diller explained that he found out that Doug Durst was trying to “wreak [havoc] on the Hudson River Park Trust because years ago the organization had spurned Mr. Durst.”

In an attempt to clear the air, Diller said he set up a meeting with Durst and demanded an explanation.

Diller recalls the conversation: “Durst said, ‘Oh no, I think what you’re doing is nice. I don’t have any objections to what you’re building. But I don’t like the fact that the Hudson River Park’s plants are only green. There’s no color and there should be and they don’t maintain it.’’”

Incredulous, Diller recalled that he said, “But that has nothing to do with my little pier. It seems awfully cruel. We’re putting up a lot of money to build this thing … He looked up at me and said, ‘You have my sympathy.’ I got up and left that room. Anyway, they’re suing us.”

Last month, Douglas Durst declined to tell The New York Times whether he was behind the litigation over Pier 55, but he did issue this statement to the paper: “I do not like the process or the project and I am in favor of the litigation.”

Melody Hahm is a writer at Yahoo Finance, covering entrepreneurship, technology and real estate. Follow her on Twitter @melodyhahm.