The definition of racism is an interesting one.
I used to just consider racism discrimination against someone based on race. But then I realised that that basic definition of racism blinds us to the nuanced historical and current systemically racist issues, which is the biggest problem.
When I look at how the colonial idea of white and black came about in the first place, it had to do with gaining resources by disenfranchising those who were considered not white. Through religious, cultural and political propoganda, non whited were classed as inferior, justifying their disenfranchisement. And this systemic racism is implememted in a complex manner that allows for plausible deniability, such as what happened in the states with redlining.
Today people see racism as basic obvious surface extremes, such as through verbal bullying, violence and blatant racism laws. To focus on this is an oversimplification, even though these are clearly a wrong thing to do. It divorces the philosophy of racism from its original intent and enforcement. Now people consider even addressing racism of the past done to a community as racism, because it has to show favouritism to a previously disenfranchised racial group in order to solve the problem, and people think that laws favouring a race at all to be racism.
To oversimplify racism in such a way is problematic. People equate verbal abuse and favourable laws towards racial groups with actual historical racial suffering of people in the past, such as enslavement and rape, lawful mutilation, destruction of homes, being treated as savages, living in poverty while so called "white" people have stolen your land and live in relative luxury, etc.
I think the lay people view vs academic view is the result of people simply not being educated or not even being interested in a topic because it doesnt affect them. I think the correct view depends on examining whoch view is more harmful vs which view is less harmful.
This is a good summary, and there's several points here worth exploring.
First, the colonialist roots of racism as it came to be practiced. At first, money was the primary motivation for the early explorations and expeditions which would lead to more colonization, particularly the Americas, where the Spanish and Portuguese got a head start, then the French jumped in to claim a share, then the British, who were somewhat late to the game. I would think they were partial towards their own nation and nationality, although the grouping and categorization of different races was probably not foremost on their minds at the beginning. The ruling monied interests would likely be interested in profit and gain from the acquisition of land and resources, and the lower classes (many of which were landless peasants) would have been attracted to the idea of open land in America. In essence, it offered a safety valve towards political stability in the homelands, as anyone who might be dissatisfied with their lot in life could then go out to a colonial territory and seek their fortune there.
Some might point to Bacon's Rebellion as a key event which solidified the idea of race-based, generational slavery. A century later when the American Revolution took place, the idea was firmly entrenched in the culture and political system - and was also integral to the economies of the Southern states. Not so much in the North, as they quickly abolished slavery after America became independent. However, that didn't make them any less racist, as they were still strong supporters of Westward expansion which involved violent relocations and atrocities motivated by racism and greed. Another aspect of Bacon's Rebellion was that poor white farmers were fighting alongside the poor black farmers, as they both were fighting the same enemy. This seemed to frighten the upper class whites who would later implement the policies mentioned. Not only was racism created to keep other races down, but it also had the effect of elevating the lowliest white peasants to a relatively higher position (even if they were still dirt poor and looked down upon by the upper classes).
The end result was that, in the perceptions of most people, there was a roughly-defined categorization of people based mainly on race. We had a certain group of people called "white," another group called "black," another called "Native American," then "Hispanics" and "Asians" (although the terminology has also evolved over the centuries). But the bottom line is that it firmly establishes a perception of a society of people grouped together into all these different groupings - like a nice tight little package - simplified and sanitized to make it easier for the public to understand.
As this has been going on for many generations, it seems that society can't really break free of this tendency to group people into their own categories. Even after the Civil Rights Movement and a period of far-reaching reforms and a genuine push towards justice and equal rights for all, people are still categorized in this way.
If people are intentionally put into groups like that, then they can often develop a group mentality and a group identity. Even people who don't want to be part of the group they've been assigned to are still identified and categorized that way just the same.