Prince Harry and Meghan Markle appear to have a very different view of what it means to be “private citizens” — and sources close to the couple are pushing back against King Charles III’s attempt to draw a clear line around their status.

The King’s recent letter made the Palace position unmistakable. Harry and Meghan are no longer working members of the Royal Family, do not represent the monarch and are not returning to official royal duties. Their position, the letter said, is “akin to private citizens with commercial and charitable interests.”

But that description has apparently not gone down well with people close to the Sussexes. The strongest response came from a source close to Harry and Meghan who told PEOPLE: “They are public figures.”

The source went even further, pointing out why the couple cannot simply be expected to disappear into ordinary private life.

“Members of the public know who they are. The idea that they can live a normal, ‘private’ life is sadly not realistic.”

And that gets to the heart of the dispute. Harry and Meghan may no longer be working royals, but they are hardly anonymous people who can walk down the street without being recognized. Their names remain globally known, their movements attract international attention, and their charitable, professional and commercial activities continue to put them firmly in the public eye.

In fact, their very return to Britain has demonstrated that reality. The Sussexes recently moved back to the UK with their children, Prince Archie and Princess Lilibet, while maintaining their home in California. Harry has also returned to a busy schedule of charitable engagements, including events connected to the Invictus Games and WellChild. Meghan continues her own professional ventures.

So while Charles can define their royal status, the argument from Harry and Meghan’s camp is that the Palace cannot simply redefine their public identity.

There is also an obvious irony in the situation. If Harry and Meghan were truly just ordinary private individuals, why would their status require a formal letter from the Lord Chamberlain in the first place?

That is essentially the argument being made by those close to the Sussexes. The letter itself was issued on Sept. 7 through Lord Benyon, the Lord Chamberlain, and clarified that Harry and Meghan remain non-working royals. It also reiterated that they should not use their HRH styles and that official and state organizations should direct questions concerning courtesies or public funding to Buckingham Palace.

For Harry, there is another major complication: security. His public profile has been at the center of his long-running legal battle over security arrangements in Britain. He has argued that his family remains exposed to risks precisely because of who they are.

That makes the phrase “private citizens” especially complicated. A normal private citizen does not typically have to worry about the security implications of being the son of the reigning British monarch, nor does a normal private citizen have an international audience following their every appearance.

Harry and Meghan may have stepped away from official royal duties, but they did not step away from fame. And they certainly did not become invisible. There was another point of friction, too.

Play

According to PEOPLE, Harry and Meghan were given details of Charles’ letter only about an hour before it was released publicly. A spokesman for the couple said they were “a little surprised not to have been told about this in advance.”

That wording is revealing. For Harry and Meghan, the message seems to have touched a particularly sensitive nerve. They may no longer carry out official royal duties, but they have never stopped being two of the most recognizable members of the Royal Family. Their lives are followed around the world, their appearances generate headlines and even the smallest decisions involving their family can become international news.

The disagreement therefore appears to be about more than titles or protocol. It is about how Harry and Meghan should be viewed now that they have returned to Britain.

Charles is making the institutional position clear: they are non-working royals and should be treated accordingly. Harry and Meghan’s side appears to be saying that there is a difference between not being working royals and being ordinary private citizens.

And that distinction may be impossible to ignore. They carry royal titles. They remain internationally recognized. They have millions of people following their lives. Harry remains closely connected to major charitable causes, while Meghan continues to build her own professional identity.

They can live privately in many aspects of their personal lives, but that does not mean the world will suddenly stop seeing them as public figures.

As the source told PEOPLE, “The idea that they can live a normal, ‘private’ life is sadly not realistic.” That may be the real message coming from the Sussex camp.

Harry and Meghan may have walked away from royal duties. But they did not walk away from the spotlight. And after returning to Britain with their children, the spotlight may be brighter than ever.