FRENCH OVERTURES

Still, France saw an opportunity to get into the game. In November 2014, DCNS CEO Herve Guillou prevailed on French Defence Minister Jean-Yves Le Drian to visit Australia and start the pitch for France.

Le Drian travelled to Albany in the country's remote southwest, where officials had gathered to commemorate the 100 year anniversary of the first sailing of Australian soldiers to fight on France's Western Front during World War One.

The poignant shared history opened the door to discussions about the submarine contract, a source close to the French Ministry of Defense told Reuters.

"The French minister wished to be there for this important event. There, he held talks with his Australian counterpart David Johnston and with ... Abbott," said the source, who along with other officials asked not to be identified because he was not authorized to speak to the media.

FIGHTING FOR JOBS

Soon after, however, Australia's political instability would erode Japan's advantage with the old guard.

In December 2014, Johnston, the Australian defence minister, was forced to resign after disparaging the skills of Australian shipbuilders.

South Australian lawmakers, worried that Abbott had quietly agreed to Japan supplying the new submarines, insisted the government look at options to build them in their state. They pressured the prime minister into holding a competitive tender which DCNS and Germany's ThyssenKrupp Marine Systems quickly joined.

In February 2015, Abbott called his "best friend in Asia", as he had previously described Abe, to tell him about the new bidding process. Abe sympathized and said he would do his best to comply, two sources with knowledge of the conversation said.

Yet, convinced the deal was still in the bag, Japan's bidding group dithered.

"Even though we were in the competition we acted as though nothing had changed," said one Japanese government source involved in the bid. "We thought we had already won, so why do anything to rock the boat?"

The Japanese did not attend a conference for the Future Submarines project in March, failing to understand the importance of the crucial lobbying event and leaving the field to their German and French rivals, sources within the Japanese bid said.

Japan's belated attempt to engage with potential local suppliers at a follow up event in August 2015 went badly.

Companies complained Tokyo was unwilling to discuss substantive deals. Having only ever sold arms to Japan's military because of a decades-old ban on exports that Abe lifted in 2014, neither Japanese company had any Australian military industrial partners.

And unlike France and Germany which quickly committed to building the submarines in Australia, Japan initially only said it would follow the bidding rules, which required building in Australia as just one of three options.

"The Japanese had been invited in on a handshake deal and were left trying to compete in an international competition having no experience in doing such a thing," an Australian defence industry source said.

By September 2015, Japan's key ally Abbott had been deposed by Malcolm Turnbull, blowing the competition wide open.

LOCAL EXPERTS

Industry officials said all of the sub offerings had some drawbacks, meaning other factors including experience and connections came into play.

Crucially, in April 2015, DCNS hired Costello, who had earlier that year lost his job as chief of staff of Australia's Defence Ministry in the wake of Johnston's resignation.

A former navy submariner who had also been the general manager for strategy at state-run Australian submarine firm ASC, Costello was ideally placed to lead a bid.

Had the Japanese called first, Costello would have likely have accepted an offer to head their bid, according to a source who knows Costello. "They didn't pick up the phone," he said. Costello declined to speak publicly about the bid.

Costello's team drew up a list of a dozen tasks DCNS needed to complete to win the deal, including the critical job of winning over U.S. defence companies Lockheed Martin Corp and Raytheon Co, one of which will eventually build the submarine's combat system.

In a final coordinated push, a huge delegation of French government and business leaders visited Australia a month ago, touting the economic benefits of their bid.

LATE JAPANESE PUSH

Finally stung to action, Japan ramped up its campaign in September 2015. Senior defence bureaucrat Masaki Ishikawa stepped in to unite what had until then been a disjointed approach spread around various ministries, the Japanese ambassador in Canberra, Sumio Kusaka, and MHI.

Japan began talking about investment and development opportunities beyond defence, including the possibility of opening a lithium-ion battery plant in Australia, while MHI opened an Australia unit.

In a last ditch attempt to woo Australia, Japan sent one of its Soryu submarines to Sydney this month. But as it sailed home on Tuesday, Turnbull announced the deal had gone to DCNS.

In an echo of his first Albany trip, Le Drian heard of France's win on Monday as he attended an ANZAC Day service for Australia's war dead in northwestern France.

For Tokyo, another big international defence competition that could help Japan develop the arms export industry that Abe envisaged is unlikely any time soon.

A more likely tack will be joint development projects with overseas partners to embed Japanese companies in military industrial supply chains. That might even include components for Australia's French submarines, one source in Tokyo said.

Other Japanese officials still want Australia to explain why they lost so they can learn from the painful and bewildering experience.

"We thought up to the end that we could have won," another source in Japan said.

(Additional reporting by Nobuhiro Kubo in TOKYO; Editing by Lincoln Feast)