This is not meant as a comprehensive answer to the question posed in the title. It is, however, what I believe to be a major contributor to the rise in independents.
Sean Theriault provided the impetus for this post with his Monkey Cage post on polarization, which is definitely worth reading in its entirety. But here are several long quotes that help explain the rise of independents:
In more recent work, I explore a second – admittedly, related dimension – of political competition. Actually, I called it “partisan warfare” in the book, but I think “political warfare” is more accurate, because partisanship is not always the cause. While it, too, may have its roots in party polarization, political warfare is more combative in nature and requires more than what can be revealed in voting patterns on the Senate floor. The warfare dimension taps into the strategies that go beyond defeating your opponents to humiliating them, go beyond questioning your opponents’ judgment to questioning their motives, and go beyond fighting the good legislative fight to destroying the institution and the legislative process. Partisan warfare serves electoral goals, not legislative goals. (emphasis added)
This warfare certainly has party polarization at its roots. Polarization may be necessary for warfare, but it is not a sufficient cause of it. Parties that are divided over policy can have a serious and honest debate, which can even become heated. . . .
This partisan warfare dimension is harder to quantify, though it most certainly exists. What I call “warfare” is what Frances Lee characterized as “beyond ideology” in her book of the same name. Lee argues that only so much of the divide between the parties can be understood as a difference in ideology. The rest of the divide–by some accounts, the lion’s share of the divide–is motivated by some other goal. I argue that it is this portion of the divide beyond ideology is what causes the angst of those participants and observers of today’s Senate. . . .
The difference between my senators is that when John Cornyn shows up for a meeting with fellow senators, he brings a pad of paper and pencil and tries to figure out how to solve problems. Ted Cruz, on the other hand, brings a battle plan.
The trick for me, and all those interested in party polarization, is coming up with systematic, repeated behaviors that differentiate ideological legislators from political warriors. The former make legitimate contributions to political discourse in the Congress; the latter don’t, and need to be called out for the havoc they wreak on our political system. The Senate has in the past and can continue in the future to accommodate senators with serious disagreements. Too many warriors in the Senate, unfortunately, will only perpetuate the dysfunction and low congressional approval we’ve seen the last couple of years.
Whether it’s called “partisan” or “political” warfare doesn’t matter to me. What’s important is that the public is affected adversely by its presence in the political system, and it’s not only evident in the Senate. It is evident in intraparty competition as well as interparty competition. Within a party, its roots are in the attempt to make the party ideologically pure, which means you identify those members of the party who are not ideologically pure enough for your tastes, and you try to eliminate them from the party’s ranks. It’s manifested in such terms as RINO—Republican in Name Only—which indicates that the person to whom it’s applied does not conform to the party’s ideology. I have observed it in attempts by Texas Republicans, supported by such groups as Empower Texans and Texans for Fiscal Responsibility, to eliminate certain members of the Texas legislature. For example, Sarah Davis (R-134) was identified by Michael Quinn Sullivan as “failing” in fiscal responsibility although her score on their scorecard was 44.7, which was higher than 23 Republican House members’ scores. The opposition to Davis is a result of her vote on HB 2 during the special session. The point is the any deviation from the party’s position results in electoral challenge and possible defeat. As the party becomes more ideologically pure, the party’s members are pushed to more extreme ideological positions. When party members become ideologically extreme, compromise with the opposition party becomes impossible, affecting interparty competition.
The result is a public that is alienated from political parties and, to a certain extent, from politics itself. The public observes a battle for survival that may be appropriate for “reality TV” but is not what the public wants from its politicians and public officials. The resulting alienation is politically damaging, especially in a democracy. The public loses faith in politics, government, and the political institutions.