Speaking in London ahead of talks with British Foreign Secretary Philip Hammond, Kerry called on the Palestinian leadership to denounce the attack, which was carried out by two Palestinians armed with a gun and axes in the ultra-Orthodox Har Nof neighbourhood.
"Our hearts go out for all Israelis for the atrocity of this event and for all the reminders of history that come with it," Kerry told reporters.
"People who have come to worship god in a sanctuary of a synagogue were... murdered in a holy place in an act of pure terror and senseless brutality and murder.
"I call on the Palestinian leadership at every single level to condemn this in the most powerful terms... and take serious steps to restrain any kind of incitement"," he added.
Signup to our newsletter and follow us on Facebook and Twitter!
"This violence has no place anywhere, particularly after the discussion that we had just the other day in Amman."
Six other people were wounded, among them two policemen, before the attackers were shot dead, police spokeswoman Luba Samri said, identifying them as Palestinians from east Jerusalem.
Palestinian Islamist group Hamas, which dominates Gaza, welcomed the attack, which took place shortly before 7 am (0500 GMT), saying it was revenge for the death earlier this week of a Palestinian bus driver from east Jerusalem who was found hanged inside his vehicle.
Police said there was no evidence of foul play, blaming his death on suicide, with their findings backed up by a post-mortem.
But colleagues and family said there were signs of violence on his body and that he was murdered.
Israeli Prime Minister Benjamin blamed the attack on "incitement" by Palestinian leader Mahmud Abbas and Hamas.