AI Has an Anti-Woman Problem. Here’s How to Fix It

Artificial intelligence has a gender bias—not because AI intentionally discriminates against women but because it reflects the biases inherent in the data on which it’s trained.

One recent United Nations study of 133 AI systems found that 44 percent demonstrated gender bias, while about a quarter showed bias for race and gender. This bias can manifest itself in the perpetuation of stereotypes around gender roles (e.g., by portraying women as homemakers), or by failing to provide information tailored to women’s unique needs. 

Chatbots designed for retirement planning, for instance, might fail to take into account women’s longer life spans and greater health care costs. Ignoring AI’s gender bias in AI could result in significant harm to women’s opportunities and financial well-being. 

AI entrepreneur Shubhi Rao is working to combat AI’s gender bias as the founder of Uplevyl, the world’s first female-forward AI startup. Her company creates specialized “vertical” AI models aimed at addressing women’s needs in the workplace, in their finances, and in their access to reproductive health. 

This transcript has been edited for length and clarity. The full interview is available at Spotify, YouTube, and iTunes. 

***

Anne Kim: Before we get to this topic of bias, let’s start with a primer on how AI large language models work. When you open up ChatGPT and a question, what is it doing to find the answer?

Shubhi Rao: These large language models—ChatGPT, Gemini, Claude, etc.,—are also called “frontier models” or “foundational models.” And I don’t think they started out wanting to build in bias. The bias is a byproduct of how the large language models have been built.

OpenAI has a beautiful story about how it started out, which was taking every single website, books, and all kinds of published material to use as the initial data set to train these algorithms. What I don’t want to take away from these frontier models is that breakthrough is huge. To see this in our lifetime is a very significant development for humankind. 

Now, of course, there’s always two sides to it. When these algorithms are trained, they are trained based on question, answer, question, answer, question, answer. And you also want a wide variety range of topics. Who does that really well is Reddit. That’s why something like 47 percent of the data sets [ingested by AI] are from Reddit because of question answer, question answer. [Another source] is Wikipedia. You go to Wikipedia, and you jump from one link to another link to another link to another link. That is also great in terms of being able to consume content in a very logical, structured way. That is something that we call building “knowledge graphs.”

So if you take a step back and at the internet for a minute, it’s very Western dominated. And if you look at any publications, whether it’s published journals in medicine or engineering or psychology or any field for that matter, men publish more than women do. 

Of course the internet reflects that, and because you ingested all that data, that’s what the frontier models do too. If you are pulling information from two baskets, and one basket is filled to the brim with data that is very skewed towards men, and the basket is only a third full of data that truly addresses the needs of women, we know that it’s going to be over indexing toward one gender over the other, statistically speaking. And that is what causes the bias. The bias is not because it was designed by the engineers to be like that. It is just the byproduct of the data sets that power these algorithms.

Anne Kim: So it’s like if you have a jar of M&Ms, and you have three bags of blue M’s and one bag of red M&Ms, you’re more likely to pull out a blue M&M when you reach into the jar. That’s basically what you’re saying.

Shubhi Rao: Correct.

Anne Kim: One of the things I did before this interview was ask ChatGPT whether AI is biased against women. And it said, “Yes, but not because AI inherently ‘hates’ women. AI systems can reproduce, or amplify gender biases that exist in their training data, design choices, and societies that produce that data.” Do you agree with that?

Shubhi Rao: That’s correct. It’s reflecting our society.

Anne Kim: What are the particular ways that mainstream AI platforms reflect this bias that you find to be particularly damaging? What have you noticed in your own work that led you to believe that this is a real problem that has to be addressed?

Shubhi Rao: Let me give you an example: workplace rights.

Women disproportionately use workplace rights because of the various life events we go through: caregiving, career breaks, many reasons. So the other day I asked ChatGPT, when you look at workplace rights, which states in the U.S. are the strongest? And it gave me California, New York, Washington—the coastal states.

I knew what the correct answer is because we do so much work in this space. And I said, “No, this is not the right answer. The number one state is Illinois.” Illinois has the strongest workplace rights. And I said, “Well, why did you make this mistake?” and ChatGPT said it made that mistake because when you look at how much content there is—not just the laws but secondary content—Illinois hasn’t necessarily produced the same level of content as California and New York have. And so it statistically thinks the coasts are strong, which they are, but Illinois is much, much stronger in the sheer number of laws and the strength of the laws as well. 

Anne Kim: I want to ask about particular categories of internet content that may be exacerbating some of these biases, like, for instance, the “manosphere.” We have thousands and thousands of hours of Joe Rogan podcasts now and that kind of thing on the internet. What kind of impact is that having on how the LLMs think, so to speak?

Shubhi Rao: It’s really interesting because when I use Claude, for example, and for somebody like me who’s a technologist—very factual, very numerical, I just want the facts, I need the data—I find it mansplaining me. So I finally just decided to write a piece of code to stop this. Although I would give it a prompt [to stop this behavior], it didn’t work. It was almost like I had to use a sledgehammer to knock it on its head to say, “Stop this. I don’t need the mansplaining.” It over-explains or it assumes that I need a lot more explanation. And it’s subtle, right? But it really bothers me.

Again, frontier models are like Google-plus. It gives you a lot of information, but you should be cautious about how you use that information. The problem is the everyday person goes into these tools wanting to get super specialized information. They don’t advertise it, and nobody should expect that. But we have set an expectation that if I go in there, it should give me accurate information. No, it’s not going do that. Just like on Google, you may or may not get accurate information. It really depends on how the ad engine worked, etc. You see the first 10 blue links, but maybe the best link you need might be on page five, right? So if we don’t expect it out of Google, why are we expecting that these frontier models will be super perfect?

Anne Kim: I want to bring up another issue that that you and I have talked about offline that I know is important to your own work, and that is the erasure of data and research involving women and gender because of the Trump administration’s campaign against “DEI.” Dataindex.us has cataloged more than 400 websites and databases that have been taken down. The Centers for Disease Control, for instance, has taken down webpages dealing with women’s reproductive health. In January 2025, it erased a dataset monitoring maternal health and infant mortality. How do you see this erasure affecting the kind of information that is available to women, but also to people generally? 

Shubhi Rao: Look, there’s over a billion internet pages.  For you and I, it hurts because we know how much content and important information lived in those 400 websites. But going back to your example of M&Ms, if you have a jar that has 100,000 red M&Ms to begin with and maybe 2,000 blue M&Ms, losing ten blue M&Ms is not going to make a material difference to the big beast at all. Mathematically, it just cannot make a big difference.

Having said that, when you take valuable content down that helps a sector of the population, that does have meaningful consequences. Why? Because over time, we will all start to shift towards building “vertical LLMs.” 

And let me give a very different example: manufacturing. My husband’s in manufacturing. He is not going to go into OpenAI to look for very specific, highly technical questions around manufacturing. They will build or they are building their own vertical LLM that encompasses all the information you need specific to manufacturing, because you have to be very precise. You can’t have a wobbly car. You can’t have wobbly airplane parts. You really need high-quality information.

Different sectors will start to build their own vertical LLMs. And so in that scenario, what Uplevyl is doing is building vertical LLMs to address issues primarily focused on gender. It would be so valuable to us to have that information from those websites. But it’s gone.

Anne Kim: Let’s turn to what Uplevyl is doing, then. What was your philosophy in starting the company, and how is your company going about tackling this problem of bias in AI?

Shubhi Rao: I wanted to build gendered data sets around the professional, personal, and financial [needs of women].

For example, in the U.S. today, we have over 91,000 jurisdictions, and our laws just around workplace rights are so fragmented. There’s federal, state, city, county, municipality, even school districts. It’s a mess. And not only that, let’s take a simple example like paid leave. Paid leave in Oklahoma may have a very different definition than paid leave in Illinois.

Let’s say you applied for paid leave, and you want to know if you’re eligible or not. It depends, right? When you start to look at the math, it’s billions of data points.

Amazing organizations fight for us to have these rights. But when these rights get published, they get published in legal language with no standardization, and the everyday person can’t even read or understand it. So then you have to go to advocates and pro bono lawyers or somebody in HR. But again, none of them have a beautiful workplace rights LLM. Such a thing doesn’t exist. That’s what we’re in the process of building. That’s one application to help people save hours, manage risk, and make it accessible to the everyday person and to enterprises. 

The second big [project] is around [women’s] financial lives. People think it’s all about budgeting, spending, and investing, but that’s just table stakes. Where the rubber meets the road is when we have to experience tough life events, like divorce, retirement, loss of a job, loss of a spouse. 

Women also live longer, and this is where a gendered data set is important. We live longer, but our healthcare costs are much higher towards the back end of our lives. So you need a very different investment profile. [This means] gendered data sets and workflows that can help an advisor help women as they navigate divorce or loss of job or any of these. 

[The third project] is focused on building around reproductive health. Reproductive health includes everything and anything related to menstruation, menopause, IVF, surrogacy, postpartum, cervical cancer—all these things that are really part and parcel of our reproductive lives.

What we’re working on is building a specialized vertical LLM to help women really navigate their lives [professionally, financially, and personally.]

Anne Kim: Just to make things more concrete, can you explain what goes into your LLM that might be different from what a standard commercial LLM may have? And how does that affect the results from a particular query? Like if I ask an Uplevyl LLM a particular question about reproductive health, how will it be different from if I ask the same question to Claude or Gemini or another commercial LLM?

Shubhi Rao: Let’s go back to workplace rights for a minute. Think of it in [three dimensions]: X, Y, and Z. So on one axis, you have to think about what jurisdiction I am in. Let’s say I am in Texas. The second is I am looking for paid leave. The third axis is my eligibility criteria, and that is why you need a workflow.

If I go into ChatGPT and say, “Can you tell me what my paid leave is?”  Well, paid leave really depends upon where your company is. And does it have more than 25 employees? Have you worked at least so many hours in a year? Have you been with the company for this much period of time? There are a lot of different criteria. And are you asking for paid leave as a caregiver, as a survivor, or for bereavement? 

ChatGPT has to give me a very personalized response. I’m not looking generically into what is paid leave. What good does that do? I need it specific to my situation and to the situation of my company. Oftentimes, ChatGPT may pull information from the state government website, but it could also pull it from different secondary sources. It can’t tell.

ChatGPT is just informational, but it’s not designed to be a workflow. That’s how a vertical LLM is different from a frontier LLM.

Anne Kim: So you’re curating the data that goes in there but then curating how that data is analyzed by that LLM so that you’re meeting the needs of the client, in which case we’re talking about women. 

Shubhi Rao: A man could use it too, but more likely it’s going to be women, yes. 

Anne Kim: What’s your advice for users of AI, like just casual users of ChatGPT or even heavy users of ChatGPT, in terms of how they can recognize bias and correct for it? Or are we kind of helpless and at the mercy of what we’re told by the frontier models?

Shubhi Rao: If it’s just something like, “Hey, give me the top ten places I should look at when I go to Seattle,” or all these little simple productivity things that save time, like a meal plan or a calendar, why not? How wrong can it be, right? And the risk is low. You have to really think about it from a risk lens.

But be super, super, super careful about providing your personal information. If you would not provide it to your colleague at work, why would you provide it to a frontier model? You don’t know where it goes, you don’t know how it’s managed. There is zero guarantee that information may not be exploited by somebody. If you don’t want any of that information to be exploited by somebody, don’t put it in. You can always use hypothetical cases, but don’t provide your own information. 

These models are also designed to sound super confident. So then people get lulled into thinking the information it gave me was perfect. Cross-check it. Sometimes you can also cross-check it between two different frontier models. You could say, I got this from ChatGPT. Gemini, critique it. Tell me why this is wrong. And so now you’ve reversed its brain, and it starts to think about why it is wrong. So if it’s important to you, do that, which is get it to cross-check.

Anne Kim: Do you have special advice for women when they’re looking at issues that may affect women more specifically? Do you have general pointers about how women can protect themselves from the overall bias of the internet and of the frontier LLMs?

Shubhi Rao: I know this sounds really terrible, but I’m a woman and woman of color, so it’s probably part of my psyche that even if I’m talking to a real human male, I am very aware that although I appreciate and respect his perspective, he perhaps doesn’t understand the world I’m coming from. I just know that he hasn’t had the same experiences as me, so by definition, he’s going to come through his own lens. 

I have the same sort of healthy relationship with these frontier models. Why wouldn’t I think of it as the same as if I were to ask another male colleague? That’s how I equate it in my brain. You should really think of it as a proxy for a male colleague. Do you think that you would get an unbiased response? Nine out of ten times the answer is going to be no. So we just have to set our expectations that the bias is going to be there. 

Unless some miracle happens tomorrow, one of two things has to happen. Lots of red M&Ms need to go away to balance the red and blue, or somehow magically, we women work to build billions and billions and billions of websites to balance the red and blue M&Ms and increase the number of M&Ms in the jar. 

The post AI Has an Anti-Woman Problem. Here’s How to Fix It appeared first on Washington Monthly.