The Hollywood star spoke at the council in her role as special envoy of the UN High Commissioner for Refugees, which has seen her visit camps hosting victims of the Syrian crisis 11 times.
An estimated 3.9 million people have fled the country in four years of civil war.
"The purpose of the UN is to prevent and end conflict, to bring countries together, to find diplomatic solutions and to save lives," said Jolie, dressed in a white blazer.
"We failed to do this in Syria."
She fumed at the stalemate, as ambassadors and ministers gathered for a debate on the humanitarian crisis tearing the country apart.
"The crisis is made worse by division and indecision within the international community preventing the Security Council from fulfilling its responsibilities," Jolie said.
The Security Council has been split between Western powers seeking to pressure Damascus and the regime's allies Russia and China, which both wield veto powers.
Jolie called for council "unity" to solve the crisis, and aid for the millions of Syrians who have fled to neighboring countries.
Some have died trying to cross the Mediterranean in rickety, overcrowded vessels to reach Europe.
Signup to our newsletter and follow us on Facebook and Twitter!
"If we cannot end the conflict, we have an inescapable moral duty to help refugees and provide legal avenues to safety," Jolie said.
The actress spoke of a "sea of excluded humanity."
"The crisis in Syria illustrates that our inability to find diplomatic solutions causes mass displacement, traps millions of people in exile, statelessness and displacement," she said.
Earlier, UN refugee chief Antonio Guterres and the head of UN humanitarian operations Valerie Amos drew a dark picture of the crisis.
"One thing is clear: the situation in the region has become utterly unsustainable," said Guterres. "There has to be massively increased support to the neighboring countries."
He noted the "growing fatigue" of host nations, and that some are even imposing harsher policies on refugees.
With some 220,000 people killed, 7.6 million displaced and 3.9 million refugees, Amos warned that "people have become numb to figures that should every day shock our collective conscience and spur urgent action."
Amos, who is due to leave her post soon, called for negotiating "humanitarian pauses," enforcing an arms embargo and targeted sanctions.
She also urged that attacks on schools and hospitals be investigated, as well as those in zones where civilians are trapped by conflict. The UN estimates that 440,000 people are in that situation.