The EU parliament motion came despite a speech by May in Florence last month offering concessions, especially on the Brexit bill, and calling for a two-year transition period after Brexit to ease the impact on citizens and businesses.
The European Parliament on Wednesday overwhelmingly backed a motion calling for trade talks with Britain to be delayed because Brexit negotiations have not made sufficient progress.
MEPs meeting in Strasbourg hit out at British Prime Minister Theresa May's government, saying infighting in her cabinet was hindering talks on key divorce issues including Britain's exit bill.
The vote called on EU leaders to postpone a decision on moving to the next phase, which they are due to make at a summit on October 19, unless there is a "major breakthrough."
Britain has called for talks on a future trade deal after it leaves the bloc in March 2019 to start now, but the EU insists the divorce must be sorted out first.
Lawmakers voted by 557 in favour with 92 against and 29 abstentions. While the vote is non-binding the parliament remains a key player as it will eventually have a veto on any Brexit deal.
Germany's Manfred Weber, the head of the largest group in the parliament, highlighted the rifts in May's cabinet by urging her to "sack Boris Johnson", the foreign minister who has undermined May on a series of key Brexit issues.
"Who should I call in London, Theresa May, Boris Johnson or even (Brexit minister) David Davis?" said Weber, a key ally of Chancellor Angela Merkel and head of the centre-right European People's Party group in Brussels.
"We need a clear answer of who is responsible for the British position."
Farage 'ransom' claim
European Parliament Brexit coordinator Guy Verhofstadt meanwhile said he was "really worried about the lack of clarity and disunity on the other side of the negotiation table."
However British eurosceptic leader Nigel Farage - who still holds a seat in the European Parliament - accused the EU of treating Britain like a "hostage" and demanding a "ransom" in the form of the exit bill.
Hours after the vote, Johnson and Davis both gave speeches at the Conservative party conference in Manchester that touched on Brexit.
Davis, who is due back in Brussels next week, said the talks were making "real steps forward" but reiterated that Britain would walk away without a deal in March 2019 if necessary.
Johnson said it was time to "let the British lion roar" and backed May's leadership.
The EU Parliament motion called on the European Council of leaders "unless there is a major breakthrough... to postpone its assessment on whether sufficient progress has been made."
It said MEPs were "of the opinion that in the fourth round of negotiations (held last week in Brussels) sufficient progress has not yet been made".
The EU's Brexit negotiating guidelines say there must be "sufficient progress" on three big issues: the bill, the rights of EU citizens living in Britain, and Northern Ireland.
EU Brexit negotiator Michel Barnier gave MEPs his assessment before the vote, agreeing that there was not enough progress yet to move to the next phase.
"There are still serious divergences, in particular on the financial settlement. We will not agree to pay at 27 what was decided at 28," the Frenchman said.
He denied Farage's accusation about holding Britain to ransom, adding: "You will never ever find the tiniest speck of vengeance or punishment in my attitude towards you. Never."
The fifth round of Brexit talks is due to start in Brussels on Monday, with negotiators needing what European Commission President Jean-Claude Juncker has called "miracles" to progress to the next stage.