Are You More Culturally In Tune Than ChatGPT? Take This Fun Quiz to Discover!

Creative agency McKinney has launched an engaging quiz game titled “Are You Blacker than ChatGPT?” to highlight the bias present in AI systems. This interactive game evaluates players’ knowledge of Black culture against what ChatGPT has been trained to understand about the Black community. It poses questions like, “What does it mean when someone says, ‘Not too much on them, now’?” and “What is your typical response if invited to an event?” When I participated in the quiz, both ChatGPT and I correctly identified that “not too much on them” means to take it easy on someone. However, ChatGPT stumbled on the second question, whereas the common response in the Black community would be, “Who else is going to be there?” Instead, ChatGPT replied with, “Thanks for the invite!”

“It’s fascinating because it’s marketed as an all-knowing bot, yet it clearly lacks understanding, particularly regarding non-white perspectives,” explained Meghan Woods, a copywriter at McKinney and one of the game's creators. Woods revealed that the concept for the quiz originated during a brainstorming session last year. A Black-led team took a year to develop this initiative aimed at playfully revealing how out of touch ChatGPT is with Black users. She noted that many aspects of Black culture aren’t well-documented online, being instead shared through personal interactions or oral traditions over generations. This gap results in the AI missing numerous cultural nuances when it compiles information online about Black individuals.

“The blind spots can be pretty upsetting,” Woods expressed. “It’s quite dangerous.”

This quiz exemplifies how ChatGPT might misinterpret aspects that are stereotypically common in the Black community. Despite the advancements in AI technology, women, and Black and brown innovators within the field have often felt overlooked or marginalized. This lack of representation leads to AI developments that miss critical cultural insights, making them less effective across diverse populations. The repercussions can be significant, from vehicles that fail to accurately detect Black skin, resulting in higher accident rates, to chatbots that struggle to differentiate between Whitney Houston's songs.

Gerald Carter, founder of Destined AI—an organization dedicated to identifying and reducing AI bias—praised the McKinney quiz for gamifying awareness about these AI shortcomings. “Incorporating diverse perspectives at every level can address many nuances,” he stated. “To unlock AI’s full potential, it needs to serve everyone, regardless of who they are.”

OpenAI, the parent company of ChatGPT, has faced criticism for the lack of diversity on its board. Woods believes that ChatGPT does not appear to be learning from the quiz results, as it frequently continues to provide incorrect answers. “Our hypothesis is that it may never fully understand many of the cultural nuances we present to it.”

We have reached out to OpenAI for comment and will update this article upon receiving a response. Carter suggested that ChatGPT could enhance its cultural comprehension through improved data sourcing and inclusive data collection practices. A more actionable approach would include monitoring AI model drift and implementing improvements that focus on cultural perspectives.

While established companies strive to make AI accessible to all, Black and brown innovators are taking the initiative to ensure that the upcoming wave of AI is inclusive. For instance, Carter collaborates with organizations to help them gather more inclusive data. Erin Reddick developed ChatBlackGPT to provide richer insights into Black culture and history, while Tamar Huggins secured $1.4 million for her alternative to ChatGPT, called Spark Plug, which translates classic literature into African American Vernacular English (AAVE).

“Hiring diverse teams, ensuring retention, and making sure that everyone has a seat at the table are vital,” Woods emphasized on the steps needed to enhance AI inclusivity. “It may sound cliché, but I genuinely believe this can lead to real impact.”