(Adds comments on Brexit, business investment)

By Rhiannon Russell

WHITEHORSE, Yukon, June 15 (Reuters) - Canadian growth is likely to be flat or slightly negative in the second quarter due to the impact of the Alberta wildfires before an outsized recovery takes hold in the third quarter, the Bank of Canada said on Wednesday.

In a speech that served as an update to the central bank's most recent forecast in April, Governor Stephen Poloz said that on the whole, the economy was making progress as it adjusted to low oil prices.

With growth expected to be "very choppy," the second and third quarters could still post an average growth rate close to the April outlook, Poloz said.

"Continued patience is required, but we have the right to be optimistic," he said. "My message is that the process has been uneven, and probably will remain so, but we are making real progress."

Nonetheless, slumping investment spending plans in the energy sector and the possibility that households will suddenly curb their spending amid high debt levels present risks to the outlook, he added.

Companies are weighing the riskiness of new expansion against the riskiness of not being able to fill their orders, Poloz said during a press conference.

"My reading of that situation is we're getting close to where that balance is tipping."

Poloz said the recent retreat in Canadian exports validated the conservative forecast the bank made in April. The sector should continue to heal, he said, helped by the depreciation of the Canadian dollar and expectations of better U.S. growth.

"While the whole process has been disappointingly slow and uneven, we remain confident that we have the right narrative," Poloz said.

Canada sank into a mild recession last year, hit by slumping crude. Although a recent recovery in oil prices was welcome, Poloz said that it would most likely not lead to greater investment in Canada's energy sector.

Asked about the upcoming referendum in Britain on whether to leave the European Union, Poloz said the bank was standing ready for possible market volatility in response to the vote.

Poloz refined his forecast on the hit from the Alberta wildfires, estimating the damage would take 1 to 1.25 percentage points off growth in the current quarter. The bank's May estimate had put the impact at around 1.25 percentage points.

The bank will update its economic forecasts at its next interest rate decision on July 13, when it is widely expected to hold rates at 0.50 percent.

(Writing by Leah Schnurr; Editing by Matthew Lewis and Sandra Maler)