Embattled Rep. Charlie Rangel (D-N.Y.) has not yet said whether he plans to seek a 21st term this year. But in upper Manhattan, a buzz about a possible open-seat race is starting to emanate.

Already Rangel has challengers: Banker Vincent Morgan, a former Rangel aide, announced his plans to take on the Congressman in the September Democratic primary. And state Assemblyman Adam Clayton Powell IV — who lost badly to Rangel in the 1994 Democratic primary and whose father was ousted from Congress by Rangel in the 1970 primary — has created an exploratory committee.

Powell has a famous name, but he’s had his share of unwanted publicity. He was pinched for drunken driving in New York in 2008 — a case that is still awaiting trial. And he dated an intern, which led to his censure by the Assembly Speaker and a new Assembly policy that forbids interns from attending receptions in Albany.

Even if Rangel decides to run again, the Democratic primary field could grow — and it would grow exponentially if he announced his retirement.

“Charlie Rangel’s really a legend back home,” said Evan Stavisky, a Democratic consultant and lobbyist in New York. “It’s certainly unseemly to challenge Rangel while he’s running for re-election again. … But if it’s an open seat, the playbook goes out the window.”

The likeliest to jump into the race with or without Rangel in it is state Sen. Bill Perkins (D). Perkins is something of a gadfly in Harlem — he was the first African-American official in New York to endorse Barack Obama for president in 2008, rather than fellow New Yorker Hillary Rodham Clinton.

Two veteran Harlem politicians with close ties to Rangel could also get into the race: Assemblyman Keith Wright (D), who doubles as chairman of the Manhattan Democratic Party, and New York City Councilwoman Inez Dickens, who is a member of Rangel’s local Democratic club. City Councilman Robert Jackson (D) is another possibility.

In an interview with the Daily News on Wednesday, Wright said “you’ve got to be brain dead not to be” considering running for Rangel’s seat if it becomes open.

But Harlem politicians aren’t the only ones who will be a factor if the Congressman retires. Rangel’s district has gone through dramatic change in recent years and takes in heavily Latino neighborhoods in Washington Heights and Inwood, as well as upper-middle class and largely white neighborhoods on the Upper West Side. According to 2007 census figures, the district population was 46 percent Hispanic, 28 percent black and 21 percent white.

So it’s highly conceivable that a Latino candidate could win Rangel’s seat: Assemblyman Adriano Espaillat (D) is seen as a leading contender, though he has been eyeing a state Senate run this year.

“This could be the long-awaited black-brown war for control of Harlem,” said one New York-based Democratic strategist who did not want to be named. “It could have lasting repercussions that ripple across New York City.”