TD Bank’s $3 billion in penalties show the high price banks can pay if they fail to comply with anti-money-laundering regulations designed to stop criminal groups from moving the proceeds of drug sales and other illicit activity through the financial system.