This weekend’s clash between supremacists and counter-protesters in Charlottesville, Va., served to highlight deep divisions in the country.

What began as a rally against the city's plans to remove a Confederate statue devolved into a deadly confrontation between demonstrators and protesters who showed up to counter their presence.

But one thing stands out in some photos and videos: the extremist white nationalists at the rally weren't dressed in robes or hoods.

The tiki-torch wielding group, made up mostly of younger men, instead donned khakis and collared shirts.

J.D. Vance, the author of "Hillbilly Elegy: A Memoir of a Family and Culture in Crisis," offered his own explanation of this “new face” of extremism on CBS This Morning on Monday, citing a rise in “identity politics” to explain the disaffection felt among the American working class.

"They’re turning to the very worst ways to solve and address that disaffection,” he said.

"One of the interesting things that I discovered just when I was doing research for the book in trying to understand the different strains of resentment that were out there is that the alt-right movement, this neo-Nazi movement is actually really driven by well-to-do, middle-class folks, people who have a good education.”

To drive home his point, Vance cited Jason Kessler, a University of Virginia graduate, who organized the weekend rally in Charlottesville.

“It’s very tempting and comforting to try to stereotype these white nationalists as a bunch of knuckle-dragging, slack jawed yokels but the truth is these are kids who are doing pretty well and are still attracted to this stuff,” he said.

Florida Holocaust Museum executive director Elizabeth Gelman disagreed with the characterization of a “new face” of extremism, saying this in a statement to 10News:

“We have witnessed people throughout the centuries try to justify their own hatred and bigotry by exploiting the fears and prejudices of contemporary society. The “new face” of fascism in America is no different. While they may refer to their narrative as competing or “alternative”, it is a narrative that continues to be manipulated specifically to prey on the fears of different segments of American society today.”

This year, Southern Poverty Law Center—which tracks hate groups—has documented 917 known active hate groups nationwide.

Currently, in Florida, there are 63 known hate groups, according to SPLC’s ‘hate map.’ The groups range from white nationalists to black separatists.

Since 2000, SPLC has documented a steady increase in known hate groups, and their data shows that growth accelerated following the election of President Barack Obama in 2008.

Among many reasons, experts who study these groups point to the rise of social media as one of the big underlying drivers that helped these groups more easily promote their message and recruit new members.

SPLC has found far right-wing populism has also driven some of the increases in these hate groups. The organization found that while the number of neo-Confederate nationalist groups—like the League of the South—increased by about 23 percent last year over the prior year, the number of known Klan chapters nationwide fell roughly 32 percent.

Among the "Unite the Right" attendees were Ku Klux Klan leader David Duke, who told reporters white nationalists were working to "fulfill the promises of Donald Trump." But Vance said he thinks the white supremacists are "a small segment" of President Trump's supporters.

"I think that's something that's important to keep in mind, is that if we want to defeat things like this, we really have to find something about our common shared purpose as Americans. And if we look at the entire swath of Trump voters and say you guys are neo-Nazi supporters, then I think we are going to be destroying some of the real cultural and social capital we need to unite as a country and actually defeat this stuff," Vance said.

On Monday, President Donald Trump responded again to the weekend violence, saying “racism is evil, and those who cause violence in its names are criminals and thugs, including the KKK, neo-Nazis, white supremacists and other hate groups that are repugnant to everything we hold dear as Americans."

The statement came after Trump faced bi-partisan criticism when his initial statements made no mention of white nationalists or neo-Nazis.

Vance said it's important for the president to name "the enemy."

"If you think about all the controversy about whether Barack Obama said 'radical Islamic terrorism,' there is a real human need for people to have their enemies named and described by their political leaders," Vance said.

Make it easy to keep up-to-date with more stories like this. Download the 10 News app now.

Have a news tip? Email, or visit our Facebook page or Twitter feed.

CBS News contributed to this report.