Cleanup efforts were set to begin Thursday at the scene of the catastrophic bridge collapse over Baltimore harbor, as authorities warned of extensive work before the major US port can reopen.

Two cranes arrived to begin the  tricky operation of clearing the twisted steel remnants of the Francis Scott Key Bridge from where it fell 185 feet (56 meters) into the Patapsco River -- blocking the entrance to the Port of Baltimore -- after being struck by a massive cargo ship early Tuesday. 

"We are moving heaven and earth" to remove the debris and get the commercial hub up and running again, senior White House official Tom Perez told MSNBC. 

Officials cautioned there would be challenges ahead, as efforts to recover the bodies of the four men still missing were called off late Wednesday when it was determined to be too dangerous to send divers into the wreckage. 

"We're... incredibly sensitive to the notion that this is also the resting place for four fathers, for four brothers, for four sons," Perez added. 

The missing men, all Latin American immigrants, are believed to have been killed when the Singapore-flagged 1,000-foot container ship Dali lost power and careened into a bridge support column. 

Nearly the entire steel structure -- crossed by tens of thousands of motorists each day -- collapsed within seconds.

The workers were part of an eight-person road repair crew working an overnight shift. Two were rescued shortly after the collapse, and two bodies were recovered Wednesday. 

Area residents attended a vigil at a nearby park Thursday morning, local media reported, while the Baltimore mayor's Office of Immigrant Affairs established a relief fund to raise money for the victims' families. 

"Our hearts are with the families," Maryland Governor Wes Moore told reporters, adding, "We are so sorry for this tragedy."

He urged patience, saying, "This work (to rebuild) is not going to take hours, this work is not going to take days, this work is not going to take weeks." 

"We have a very long road ahead of us."

'Substantial loss'

The Federal Highway Administration said it would honor the Maryland Department of Transportation's request for an initial $60 million for what Moore called "immediate response efforts, and to lay the foundation for a rapid recovery."

"The federal government is committed to providing all necessary resources to rebuild the bridge," the agency's administrator Shailen Bhatt said in a statement. 

The disaster could result in the largest marine insurance payout ever, according to the head of insurance giant Lloyd's of London, Bruce Carnegie-Brown. 

"It feels like a very substantial loss, potentially the largest-ever marine insured loss, but not outside parameters that we plan for," he told CNBC. 

The harbor's closure also raised concerns for the local economy -- with 140,000 jobs supported by the port -- and the wider national supply chain.

Baltimore is the biggest vehicle-handling port in the country, including cars and heavy farm equipment, according to US Transportation Secretary Pete Buttigieg. About $100 to $200 million in value comes through the port daily.

Up the coast from Baltimore, the Port Authority of New York and New Jersey will plan take on additional cargo to help blunt the supply chain impacts, the governors of those states pledged in a joint statement Thursday.