More than half the catch brought ashore at the Belgian port of Ostend is from British waters. Unless the UK agrees to stick to EU fishing rules after Brexit, all this could be lost.
The British government has announced it's pulling out of the London Fisheries Convention.
The pact has allowed European fishermen to catch fish in UK waters since 1964.
Environment minister Michael Gove says Britain will take back control of its own waters after Brexit.
Fishermen in Belgium say the decision could cost them their livelihoods.
"60 percent of the total, we fish in British waters. We hope we can make a good agreement, so that we can still fish there, and that is good for both sides," says Bruno Decordier, who has been fishing in the North Sea since 1986.
Elena Casas reports from the Belgian fishing port of Ostend.