In general, I think this is a risk you have to take with any democracy. In the United Kingdom, you had a single direct democratic vote basically wreck the country's foreign policy through Brexit, but in the United State Donald Trump didn't even win the popular vote and he did a whole lot more than just set back our foreign policy.
I think the jury-type system would do a lot to ameliorate these effects. Make someone spend an hour talking to a panel of experts and eleven strangers from across the political spectrum, and they're going to talk, think, and vote a lot differently than they would stuck in their own bubble.
I know there's a Youtube channel where they get three people on polar opposites of an issue to sit in a room together and discuss, and I think they're usually able to reach common ground. People are social animals after all. I'm sure there are other solutions people have come up for this, I haven't taken the time yet to look specifically into deliberative democracy.