AI came for beer. Judges got pissed

Hey there, Joseph here. So I love craft beer. I especially love the human aspect of drilling down what exactly you, and maybe only you, can taste in a specific beer. Well, a prominent beer competition introduced an AI-judging tool without warning earlier this year. As you might expect, judges were not happy. The months-long episode included open letters, threats of legal action, and judges talking to me anonymously to protect them from retaliation. Maybe grab your favorite drink—whatever that might be—and read on to see what happens when someone tries to put AI on something very, very human. As one judge told me: “brewing is an art.”

A prominent beer judging competition introduced an AI-based judging tool without warning in the middle of a competition, surprising and angering judges who thought their evaluation notes for each beer were being used to improve the AI, according to multiple interviews with judges involved. The company behind the competition, called Best Beer, also planned to launch a consumer-facing app that would use AI to match drinkers with beers, the company told 404 Media.

Best Beer also threatened legal action against one judge who wrote an open letter criticizing the use of AI in beer tasting and judging, according to multiple judges and text messages reviewed by 404 Media.

The months-long episode shows what can happen when organizations try to push AI onto a hobby, pursuit, art form, or even industry which has many members who are staunchly pro-human and anti-AI. Over the last several years we’ve seen it with illustrators, voice actors, music, and many more. AI came for beer too. 

“It is attempting to solve a problem that wasn’t a problem before AI showed up, or before big tech showed up,” Greg Loudon, a certified beer judge and brewery sales manager, and who was the judge threatened with legal action, said. “I feel like AI doesn’t really have a place in beer, and if it does, it’s not going to be in things that are very human.”

“There’s so much subjectivity to it, and to strip out all of the humanity from it is a disservice to the industry,” he added. Another judge said the introduction of AI was “enshittifying” beer tasting.

This story started earlier this year at a Canadian Brewing Awards judging event. Best Beer is the company behind the Canadian Brewing Awards, which gives awards in categories such as Experimental Beer, Speciality IPA, and Historic/Regional Beers. To be a judge, you have to be certified by the Beer Judge Certification Program (BJCP), which involves an exam covering the brewing process, different beer styles, judging procedures, and more.

Around the third day of the competition, the judges were asked to enter their tasting notes into a new AI-powered app instead of the platform they already use, one judge told 404 Media. 404 Media granted the judge anonymity to protect them from retaliation.

Using the AI felt like it was “parroting back bad versions of your judge tasting notes,” they said. “There wasn't really an opportunity for us to actually write our evaluation.” Judges would write what they thought of a beer, and the AI would generate several descriptions based on the judges’ notes that the judge would then need to select. It would then provide additional questions for judges to answer that were “total garbage.”

“It was taking real human feedback, spitting out crap, and then making the human respond to more crap that it crafted for you,” the judge said.

“On top of all the misuse of our time and disrespecting us as judges, that really frustrated me—because it's not a good app,” they said.

Screenshot of a Best Beer-related website.

Multiple judges then met to piece together what was happening, and Loudon published his open letter in April.

“They introduced this AI model to their pool of 40+ judges in the middle of the competition judging, surprising everyone for the sudden shift away from traditional judging methods,” the letter says. “Results are tied back to each judge to increase accountability and ensure a safe, fair and equitable judging environment. Judging for competitions is a very human experience that depends on people filling diverse roles: as judges, stewards, staff, organizers, sorters, and venue maintenance workers,” the letter says.

“Their intentions to gather our training data for their own profit was apparent,” the letter says. It adds that one judge said “I am here to judge beer, not to beta test.” 

The letter concluded with this: “To our fellow beverage judges, beverage industry owners, professionals, workers, and educators: Sign our letter. Spread the word. Raise awareness about the real human harms of AI in your spheres of influence. Have frank discussions with your employers, colleagues, and friends about AI use in our industry and our lives. Demand more transparency about competition organizations.”

33 people signed the letter. They included judges, breweries, and members of homebrewer associations in Canada and the United States.

Loudon told 404 Media in a recent phone call “you need to tell us if you're going to be using our data; you need to tell us if you're going to be profiting off of our data, and you can't be using volunteers that are there to judge beer. You need to tell people up front what you're going to do.”

At least one brewery that entered its beer into the Canadian Brewing Awards publicly called out Best Beer and the awards. XhAle Brew Co., based out of Alberta, wrote in a Facebook post in April that it asked for its entry fees of $565 to be refunded, and for the “destruction of XhAle's data collected during, and post-judging for the Best Beer App.”

“We did not consent to our beer being used by a private equity tech fund at the cost to us (XhAle Brew Co. and Canadian Brewers) for a for-profit AI application. Nor do we condone the use of industry volunteers for the same purpose,” the post said.

Ob Simmonds, head of innovation at the Canadian Brewing Awards, told 404 Media in an email that “Breweries will have amazing insight on previously unavailable useful details about their beer and their performance in our competition. Furthermore, craft beer drinkers will be able to better sift through the noise and find beers perfect for their palate. This in no way is aimed at replacing technical judging with AI.”

With the consumer app, the idea was to “Help end users find beers that match their taste profile and help breweries better understand their results in our competition,” Simmonds said.

Simmonds said that “AI is being used to better match consumers with the best beers for their palate,” but said Best Beer is not training its own model.

Those plans have come to a halt though. At the end of September, the Canadian Brewing Awards said in an Instagram post the team was “stepping away.” It said the goal of Best Beer was to “make medals matter more to consumers, so that breweries could see a stronger return on their entries.” The organization said it “saw strong interest from many breweries, judges and consumers” and that it will donate Best Beer’s assets to a non-profit that shows interest. The post added the organization used third-party models that “were good enough to achieve the results we wanted,” and the privacy policies forbade training on the inputted data.

A screenshot of the Canadian Beer Awards' Instagram post.

The post included an apology: “We apologize to both judges and breweries for the communication gaps and for the disruptions caused by this year’s logistical challenges.”

In an email sent to 404 Media this month, the Canadian Brewing Awards said “the Best Beer project was never designed to replace or profit from judges.” 

“Despite these intentions, the project came under criticism before it was even officially launched,” it added, saying that the open letter “mischaracterized both our goals and approach.”

“Ultimately, we decided not to proceed with the public launch of Best Beer. Instead, we repurposed parts of the technology we had developed to support a brewery crawl during our gala. We chose to pause the broader project until we could ensure the judging community felt confident that no data would be used for profit and until we had more time to clear up the confusion,” the email added. “If judges wanted their data deleted what assurance can we provide them that it was in fact deleted. Everything was judged blind and they would have no access to our database from the enhanced division. For that reason, we felt it was more responsible to shelve the initiative for now.”

One judge told 404 Media: “I don’t think anyone who is hell bent on using AI is going to stop until it’s no longer worth it for them to do so.” 

“I just hope that they are transparent if they try to do this again to judges who are volunteering their time, then either pay them or give them the chance ahead of time to opt-out,” they added.

Now months after this all started, Loudon said “The best beers on the market are art forms. They are expressionist. They're something that can't be quantified. And the human element to it, if you strip that all away, it just becomes very basic, and very sanitized, and sterilized.” 

“Brewing is an art.”