While Belarus manufactured the refugee crisis situation, Poland shares responsibility for the acute suffering in the border area, the Human Rights Watch said.
While Belarusian border guards may be guilty of torture against refugees on the country's border with Poland, both nations have committed "serious human rights violations".
Human Rights Watch said in a report on Wednesday that the two governments "have an obligation to prevent further deaths by ensuring regular humanitarian access to the people stuck in the border area".
HRW researchers said they had conducted in-depth interviews with 19 people, some of whom "had been pushed back, sometimes violently, by Polish border guards".
They said these pushbacks "violate the right to asylum under EU law" and urged the EU to "start showing solidarity with the victims at the border on both sides who are suffering and dying".
In Belarus, they said that "violence, inhuman and degrading treatment and coercion" by Belarusian border guards were "commonplace".
HRW added that this treatment "may in some cases constitute torture, in violation of Belarus's international legal obligations".
"While Belarus manufactured this situation without regard for human consequences, Poland shares responsibility for the acute suffering in the border area," Lydia Gall, senior Europe and Central Asia researcher at HRW, said in a statement.
The report also said that three people had accused Polish border guards of separating their families, including parents from children, by taking those in need of medical care to the hospital while sending the other family members back to Belarus.
It urged Belarus and Poland to "halt ping-pong pushbacks and allow independent observers, including journalists and human rights workers, access to currently restricted border areas".
The EU and US have accused Belarus of engineering the crisis in retaliation against EU sanctions on President Alexander Lukashenko, who has crushed the opposition and independent media during his nearly three decades in power.
Lukashenko has told the BBC that Belarusian border guards "maybe" helped the refugees cross, but denied bringing them to Belarus and urged the EU to take them in.
Belarus estimates there are around 7,000 refugees in the country.