The legal challenge comes amid growing international criticism of Israel for its use of lethal force and a mounting casualty toll since the weekly protests started.
Israeli and Palestinian rights groups went to Israel's Supreme Court on Monday in a bid to stop the army's use of lethal force against Palestinians during protests and clashes on the Gaza border.
As many as 48 Palestinians have been killed by Israeli fire on the frontier since the start of protests that organisers have dubbed the Great March of Return on March 30, with more than 1,500 wounded.
No Israelis have been hurt.
The hearing came amid growing international criticism of Israel for its use of lethal force and a mounting casualty toll since the weekly protests, organised by Gaza's ruling Hamas, began in late March. Organisers say the mass demonstrations are to continue for at least two more weeks, with some threatening a mass border breach.
No ruling before next week
The court is not expected to rule before next week, in what human rights lawyer Michael Sfard said is the first broad review of the army's rules of engagement in almost three decades.
On Monday, six human rights groups asked the Supreme Court to declare as unlawful any regulations that allow soldiers to open fire at unarmed civilians.
Lawyers for the groups said Israel's response to the Gaza protests must follow the rules of engagement for law enforcement officers, who are barred from using lethal force unless they face imminent danger to "life and limb."
The Israeli military argued that the protests are taking place in the context of a long-running armed conflict with the militant group Hamas and that open-fire regulations are subject to the rules of armed conflict. Such rules provide greater leeway for the use of lethal force than those governing law enforcement practices.
Sfard said the army's rules of engagement don't meet international standards of law enforcement and that the laws of armed conflict don't apply in this case. "Lethal force against unarmed civilians who do not pose danger is illegal," he said. "This is the crux of the case."
Michael Oren, a deputy Israeli Cabinet minister, said the protests "are designed to break down the border and any army is charged with defending a border." He acknowledged that mistakes might have been made in some of the shootings and that "bullets can do unpredictable things."
The weekly marches are aimed in part at trying to break a border blockade of Gaza, imposed by Israel and Egypt after Hamas overran the territory in 2007, a year after winning Palestinian parliament elections. Organisers say the marches also press for the "right of return" of refugees and their descendants to what is now Israel.
The protests, dubbed the "Great March of Return," are to culminate in a mass gathering on the border on May 15, a day Palestinians mark as their "nakba," or catastrophe, to commemorate their mass uprooting during the 1948 war over Israel's creation.
Hamas leaders have issued warnings about a possible border breach, but stopped short of specific threats. "Our steadfast national decision is to continue the March of Return and of breaking the siege," Hamas leader Ismail Haniyeh said Monday. He said he hoped the idea would catch on in the West Bank and elsewhere.
In the weekly protests, thousands of Palestinians have been heading toward Gaza's border area with Israel. A majority stay in or near five tent camps, each set up several hundred metres from the border fence. Typically, smaller groups move closer to the fence, throwing stones, burning tires and or hurling firebombs.
Soldiers, including snipers, are perched behind protective sand berms on the other side of the border.
Last Friday, hundreds of Palestinians converged on a stretch of fence, trying to burn and rip through it before drawing heavy Israeli fire. Three Palestinians were killed and dozens were wounded in the incident, the most serious attempt so far by a large group to break down the barrier.
The rights groups that petitioned the high court said even if protests turn into riots, lethal force against unarmed demonstrators can only be used a last resort — in the event of a life-threatening situation. The groups said in several cases caught on video, protesters were targeted while standing dozens of metres from the fence or while trying to take cover. Among those killed were four minors and two journalists.
The Israeli military said Hamas has used the protests as cover to damage the fence and carry out attacks. On Sunday night, Israeli troops fatally shot two Palestinians who infiltrated from Gaza and attacked soldiers with explosives, the military said. A third Palestinian was killed during an attempted border breach, it said.
In its court brief, the army said Gaza militants have planted some two dozen explosives along the fence.