(C) Daily Kos This story was originally published by Daily Kos and is unaltered. . . . . . . . . . . The Downballot: When is a poll a quality poll? with Natalie Jackson (transcript) [1] ['Daily Kos Staff', 'Backgroundurl Avatar_Large', 'Nickname', 'Joined', 'Created_At', 'Story Count', 'N_Stories', 'Comment Count', 'N_Comments', 'Popular Tags'] Date: 2023-02-09 This transcript has been lightly edited for clarity. David Beard: Hello and welcome. I'm David Beard, contributing editor for Daily Kos Elections. David Nir: And I'm David Nir, political director of Daily Kos. The Downballot is a weekly podcast dedicated to the many elections that take place below the presidency from senate to city council. Please subscribe to the Downballot on Apple Podcasts and leave us five-star rating and review. David Beard: What are we going to be covering on today's show? David Nir: We, as always, have a lot to talk about. There were three big special elections for the Pennsylvania State House that saw Democrats turn in a thunderous performance. Then in Wisconsin in two key races, Democrats and progressives are doing everything they can to pick their preferred GOP opponents in the upcoming primaries. In Indiana, we saw the first true House retirement of the cycle when a second-term Republican announced that she was done. And finally, we are going to be talking about a proposal to expand the size of the House of Representatives, which has been capped at 435 for more than a century. After our weekly hits, we are going to be talking with Natalie Jackson, who is Director of Research at PRRI, about polling in the 2022 elections, polling aggregators, and what makes for an actual high-quality poll. Beard, we had some big special elections in Pennsylvania on Tuesday. Tell us all about them. David Beard: Well, it looks like the Pennsylvania state House might finally get some resolution. Now, the Democrats won three landslide victories in special elections, as you mentioned, on Tuesday. Now Democrats won a 102 to 101 edge back in November, but Republicans sort of temporarily enjoyed a 101 to 99 advantage in terms of the actual members in the chamber because three Democratic held seats were vacant. Two members had resigned to take higher office and one member had died just before the election. But all three Democratic candidates in these specials — Joe McAndrew, Matt Gurgley, and Abigail Salisbury — all won their seats all in really impressive fashion, which means when the chamber reconvenes Democrats will actually have 102 members and be able to actually control the chamber. And these weren't just wins, but they were big overperformances compared to Biden's results in 2020. One of them was actually the biggest special election overperformance of Biden 2020 numbers that we've seen so far since 2020. Matt Gurgley, one of those three candidates won his seat 74% to 25% in a seat that Biden had only taken 58% to 41%. That's a 32-point overperformance in this special election, which is just astounding compared to what we usually see, and when you think about special elections can often be more favorable for the out party that doesn't hold the presidency. That's just a crazy, crazy result. Now while moderate Democrat Mark Rozzi had been elected Speaker earlier in the year with the support of a small number of Republicans, he's really managed to upset folks on both sides of the aisle who had originally voted for him. So it's not exactly clear what's going to happen when the chamber reconvenes, but Democratic leader, Joanna McClinton, who was previously expected to become speaker, she said on Tuesday night after the election results, and I quote, "Please stay tuned to see what the will of this body will be on the date that we return to the voting session." So that's clearly an expectation that something's going to be happening once they've got 102 Democrats back in that chamber. David Nir: One other thing I should note about those specials, Beard, is that turnout was really high, shockingly so. There was a special election in a neighboring district — these were all in the Pittsburgh area — just last year, and something like 5,500 people showed up to vote. In these elections, you had more like 10 to 12,000 people showing up to vote. And that's really remarkable to me because in these blowouts, people usually feel less incentive to show up. And if anything, it might have felt confusing for Democratic voters because didn't we already win the majority in November? But no, the messaging clearly got through about how important these races were and Democrats really showed up. And if that continues for the rest of this cycle, watch out Republicans. David Beard: I think one thing that you often see with these sort of blowout numbers in non-competitive special elections is it's usually because the other party doesn't bother showing up, but what we saw here is you said with this massive turnout, was Democrats really turning out. If that's because the message about the majority being on the line came through or just continued energy from 2022, either way, it's definitely a good early sign. David Nir: I have a question about Mark Rozzi. Do you think he's sort of like the Liberal Democrats of the Pennsylvania state House? The liberal Democrats in the UK really kind of screwed themselves several years back by aligning with the Tories and they weren't conservative enough for Tory voters and then they ruined their reputation with more middle of the road voters. Rozzi kind of said or supposedly said he was going to govern as an independent, and then he Republicans claimed he went back on that because he never changed his party registration feels like he's just caught in the middle in a bad way, and it may be his own doing. David Beard: Yeah, I think he's somebody who saw the opportunity to become Speaker of the Pennsylvania House and didn't really have a plan for the next day. And we saw that play out because he became Speaker and then disaster sort of happened one after another. There was clearly no plan for it. He didn't have some sort of big idea to make this work. And so it would not surprise me if this was very short-lived. David Nir: We're going to change gears and talk about Wisconsin right now, and this is one of my absolute favorite topics in politics. We have two huge races coming up in April in Wisconsin, one for the state Senate and an even bigger one for the state Supreme Court. And in the past week in both races, progressives have taken major steps to shape the race by picking their GOP opponents. In the state Senate race, Jodi Habush Sinykin is the only Democrat running there. Several Republicans in the race, including two members of the state Assembly. And Habush Sinykin has started running ads calling state Rep. Janel Brandtjen, quote, "The most conservative you can be when it comes to abortion." And that obviously looks like an attack coming from a Democrat. But of course that's a message that's designed to be music to the ears of GOP primary voters. And Democrats would very much prefer to face Brandtjen on April 4th rather than her top rival, another member of the state Assembly, Dan Knodl. And in fact, outside Republican groups are spending to boost Knodl in the primary, which is on Feb. 21, so less than two weeks away. So why is Brandtjen so awful? Well, she's an election conspiracy theorist, but her real sin was supporting a Trump-backed primary challenge to the Speaker of the Assembly Robin Vos. Vos wound up winning very narrowly then he won another term in the fall and became Speaker again. And at that point Republicans said they'd had enough of Brandtjen and actually kicked her out of their caucus meetings. She is total persona non grata. She has a reputation as just truly Looney Tunes and there's no question that she would be the easier candidate for Habush Sinykin. And that really matters because a lot is on the line here. We've mentioned this race before. Republicans just won a supermajority in the state Senate in November. If Democrats were to flip this seat, they would undo that supermajority. And this is a tough district. This is a district that Trump won by five. Democratic governor Tony Evers lost it by four points last year in his successful reelection bid. So Democrats need to try to find every advantage they can get. And I want to point out something that I think is really, really important. There was so much handwringing over Democrats pursuing this hardball tactic in 2022. Here are the facts. It worked perfectly. Daily Kos Elections has been tracking these kinds of activities very, very closely. And there were 17 races across the country where Democrats tried to pick their GOP opponent. They were successful in eight of them. In all eight of those races, Democrats went on to win the general election and the pearl-clutchers fretted about how this would represent a possible blow to democracy if these candidates that Democrats are elevating actually go on and win. The fact of the matter is whether it's Knodl or whether it's Brandtjen, if either of them wins the general election, Republicans will still have a supermajority in the Senate. That GOP Senate leadership in the state of Wisconsin will still suck either way. But if Habush Sinykin wins, then boom, that supermajority is gone. And that is huge for democracy. And I'll also add this, that ad that Habush Sinykin is running right now, she could run the exact same ad in the general election if Brandtjen is the nominee. It's really amazing how the very same messages that are appealing to Republican voters in a primary are absolutely just radioactive for regular voters in a general election. That is a huge, huge problem for Republicans. David Beard: And the thing that I always emphasize when we talk about this is that Republican primary voters have agency here. If they think that it's better to vote for Knodl because he has more likely to win, they are free to do so. They are the ones in those eight races where Democrats got the candidate that they wanted and then beat them. The Republican primary voters are the ones who voted for those candidates to advance to the general election. So as long as that's going to keep happening and the Democrats can beat them in the general, Democrats are going to keep doing it. David Nir: So the situation is actually a bit different in the other race. The one for the Supreme Court. That's an officially nonpartisan contest. You don't run under party labels, but it's very clear who the progressive candidates are and who the conservatives are. And in that race, a progressive group just started running ads slamming conservative Judge Jennifer Dorow for giving light sentences to criminals including sex predators during her career on the bench. And these ads are obviously quite different from the abortion ads that are being run in the state senate race. This is just a straight up attack and either it will weaken Dorow in the primary or it'll just soften her up for the general election. Progressives would rather face former Justice Dan Kelly. He was an appointee of former Governor Scott Walker. Kelly badly lost a bid for a full term in 2020 on the Supreme Court. Dorow, by contrast — very unusually for a judge — she has a lot of name recognition because she presided over the trial of the man who was ultimately convicted of killing six people at the Waukesha Christmas parade in 2021. And just to reiterate, as we've mentioned on the show before, this is a top two primary. So the two highest vote getters will advance to the general election. Again, there are no party labels. So you could have a situation where two conservatives advance to the general election on April 4th. That would be obviously a nightmare. Conceivably two liberals could advance. I think that's probably pretty unlikely. Right now progressive Judge Janet Janet Protasiewicz is the only liberal who is on the airwaves. She is far out spending all of her other opponents, so I think she's very likely to advance the general election. The real question is whether it'll be Kelly or Dorow? David Beard: And as we've talked about this race before, but it is one of the most important races of 2023. Like you said, I expect this to come down to a progressive versus conservative candidate for the April 4th election. And the result of that race determines the control of the Wisconsin Supreme Court. It determines what sort of rule of law the state of Wisconsin is going to be following when that legislature decides to do whatever it wants, and whether or not there's a check on it in the Supreme Court. So this is a huge, huge race that will continue to follow up through April. Now in the U.S. House, we actually had our first official full retirement when Indiana Representative Victoria Spartz announced on Friday that she wouldn't run for reelection or for any other office in 2024. Now, we've had some other members declare for higher office, the way that Katie Porter and Adam Schiff are running for Senate in California, but this is the first Representative to just announce that she was retiring and not running for anything. This opens up the 5th District in Indiana, which was competitive back in 2020, but was gerrymandered by the Indiana Republicans to be much safer, took in a bunch more Republican turf. So whoever wins the Republican primary there will almost certainly win the general election in 2024. As I mentioned back in 2020 when Spartz first won the seat, it was a very competitive race. She defeated former State Representative Christina Hale by only a 50-46 margin. So that was really seen as a seat that was trending Democrats' way. This is the northern Indianapolis suburbs. It took in a bunch of redder, more rural areas to make it safer for the GOP, and so probably the winner of the GOP primary is likely going to win the seat in 2024. Now, by also deciding not to run for Senate, Spartz has further cleared the field for far-right Representative Jim Banks, who's the strong favorite for the open Senate seat there. We discussed that in a little more detail last week. David Nir: Even though Republicans tried to shore up this district for Spartz and for whoever might succeed her, I think it's still very probable that this area is trending toward Democrats. And maybe not in 2024, but possibly by the end of the decade, we could see this seat come into play once again, as we've seen in other suburban areas where blue tendrils keep spreading further and further outside. And the other thing I want to add is Spartz has been in Congress for just a shade over two years. She won her second term in November, so that means she really only served one full term and she's already calling it quits. Now, you never know why one individual member decides that they want to be done with Congress, but we've seen this a number of times over the last decade where a Republican member of the House leaves after a very short tenure. I think a lot of them really just don't like being there, and it's all the more remarkable given that they just won the majority back. David Beard: Yeah, you'd think being in the majority would be a good incentive to stay. We often see a lot of members who enter into the minority then retire for the upcoming cycle, particularly if it seems like there's no chance of winning that majority back quickly. Obviously it's incredibly early in the cycle, but that's something that we haven't seen on Democrats side yet. I do think there's a big belief within the Democratic Party that the House can be won back in 2024, so I think that may keep a lot of Democrats sticking around who might have otherwise retired if they were looking at six, eight, or ten years in the minority. David Nir: Speaking of the House, I want to wrap on a topic that we haven't ever addressed before on the show, and that is the size of the chamber. With one minor exception, the United States House of Representatives has been capped at 435 members since 1913. So we are talking now 110 years. And in that time, the country's population has quadrupled, and that means that the number of people that each member of the House represents has also shot up dramatically. Oregon Representative Earl Blumenauer has introduced a bill that would rectify this problem by increasing the size of the House to 585 members. So this bill would not take effect until after the 2030 census and the round of reapportionment that would take place then. But we ran the numbers to see what would happen with the 2020 census data, and you can find that. We will include a link in our show notes, and you can also see it pinned to the top of our Twitter account. That's @DKelections. Almost every state would gain representatives, except for the very smallest. California, unsurprisingly, as the biggest state, would see the largest gain. It would shoot up from 52 seats to 70 seats. And there are a lot of potential upsides to increasing the size of the House. There would be more opportunities to elect candidates of color, first and foremost, and also constituent services would be improved. Because right now, members of Congress have so many constituents who reach out to them for help. If they had fewer, they would be able to serve people in need much more quickly. Now, there are some potential downsides. A bigger House would almost certainly concentrate even more power among party leaders, but that's part of an ongoing trend anyway. The other thing that's worth mentioning is a lot of people hold out the idea of a larger House as an antidote to gerrymandering. They think that more members and smaller districts would make it harder to gerrymander. That really isn't the case, and we know this because so many state legislatures, which already have much smaller districts, are already terribly gerrymandered. In fact, the biggest state legislature in the country, which is the New Hampshire House, which has 400 members in a state with a population of 1.4 million. So each member only represents a few thousand constituents. Republicans gerrymandered the hell out of the New Hampshire State House. So really it's not a cure all for that, but it would very likely make government more responsive. Now, the fact of the matter is Blumenauer's bill isn't likely to come up for a vote anytime soon. A key reason is that politicians simply hate having their influence reduced. If California had 70 members of the House instead of 52, each incumbent would be proportionately less important and influential. But if we don't do something, then under current trends, the average U.S. House district will in the coming decades have more than a million members. It's a really ridiculous figure, and it's also very far outside of international norms. Political scientists who've studied this have found an interesting relationship. It turns out that in the lower House of Parliament in most countries, the number of members is pretty closely proportional to the cube root of the country's population. If the United States followed that pattern, we'd actually have about 690 members, so 100 more than even Blumenauer is proposing. But it's important to talk about this topic because if we don't, then we will never see change. David Beard: Absolutely. And it is one of those situations where any changes will probably need to be bipartisan because of the difficulty in passing laws in general and the way that you could run against this, talk about creating more politicians, creating more DC elite jobs, et cetera, et cetera. So whether or not you can find some Republican support for this idea, that remains to be seen, but it's certainly worth putting out there, pushing, seeing where you can go with it. David Nir: Well, that's going to be a wrap for our weekly hits. Coming up, we are talking with Natalie Jackson, who is the director of research at PRRI to share her thoughts about polling in the 2022 election and the future of polling aggregation. Please stay with us. I am very excited for our guest today. We are joined by Natalie Jackson, who is director of Research at PRRI, that is the Public Religion Research Institute. Natalie, thank you so much for joining us today. Natalie Jackson: Yeah, thanks for having me. David Nir: So your work is in the area of polling, and you have an academic background, but you also publish a lot of stuff for a wider audience. You're a big presence on Twitter. I'd love to hear about how you got into this world of polling and statistics. Natalie Jackson: Sure. I got into the polling field kind of accidentally. I think there are quite a few of us in the field that have similar kind of stories where we didn't really realize it was a job thing, sort of career thing, but we ended up in it anyway. So, when I finished my undergraduate degree, I was applying to PhD programs, and the entire idea was that I wanted to teach. There was no other goal really than just teaching college. This was 2006; higher ed was still a pretty good job market. Instead of getting assigned as a teaching assistant though, when I got to the University of Oklahoma, I was assigned as a research assistant at the on-campus Survey Center. So like a lot of universities, the University of Oklahoma had, at that time, a Survey Center doing public opinion projects for the state of Oklahoma, for different interest groups at the University, just state level interest projects that were based on surveys. And so I ended up working as a graduate assistant running surveys from the time I started my PhD. I was actually very upset about this because I wanted to teach, which is very funny now. Literally, I was sobbing crying when I didn't get a teaching assistantship. But over time, I really fell in love with the process of talking to people and then analyzing the data and then reporting the data. So along the way, my supervisor had taken me to the American Association for Public Opinion Research conferences. And so when I was finishing my PhD, I did a postdoc at Duke and I missed doing surveys, so let me go do surveys. I was really fortunate. The first job I got after my postdoc was at the Marist Poll, working with the great crew up there, and so that's where I really started cutting my teeth in political polling. We had the contract that year to do state level polling for NBC and the Wall Street Journal. This was the 2012 cycle, and I was a junkie. I was caught. That was it. David Nir: You were hooked? Natalie Jackson: Yep, I was gone. And then I worked at HuffPost for a little while and got really hooked on the communication side of it, explaining things and general audiences, and really just putting my voice out there to try to be a voice of reason in all the clutter. Again, I was just hooked on the way the whole industry connects between journalism and polling. And so, yes, I have this academic background. It's really useful, but I do love the practical work. David Nir: And so now you are at PRRI. You're the Director of Research there and a lot of folks may not be familiar with your organization. We love your data and rely on it all the time, but I'd love for you to tell us a little bit about what your group does. Natalie Jackson: PRRI is a really interesting place to be right now. So the mission of the organization is to study the intersection of religion, culture, and public policy. So our founder, Robert Jones, started PRRI more than a decade ago with the idea of shining more of a light on how religion operates in our society and within our politics. So what we really focus on is all the things that come together that impact people's lives and that result in their opinions on various policy issues. We're a small operation. We do a lot of projects around specific policy issues like immigration, LGBTQ rights, abortion. We have a larger research portfolio around racial justice, white Christian nationalism. We've done a lot of work on Qanon. So basically it's where there's a heavy infusion of religious influence into our political and policy world. And all the work we do is public opinion. It's all surveys. We do really the best quality surveys we can do. We're not doing inexpensive things. We're doing large sample sizes off of well-known vendors like Ipsos. They're our primary partner. And the idea is just to put out rigorous, good quality public opinion data on how all of the things swirling around in people's lives impact them and shape their views. David Nir: You mentioned abortion, which has really been the number one topic on this podcast for getting on close to a year now. I find this a fascinating topic when it comes to religion specifically because as a Jew when I hear about the views of religious folks on abortion, that often tends to be a shorthand for a particular type of Christian outlook. And Jewish beliefs on abortion tend to be very different. I'd be really curious to hear what you have found in terms of breakdowns among different religious groups and their views on abortion and its legality in this country. Natalie Jackson: There are a lot of differences between religious affiliations and their views on abortion. Of course, the real outlier is white Christian evangelical group. They are the group that's the most opposed to legal abortion. That shouldn't surprise anyone. Jewish people, religiously unaffiliated people, other non-Christian religions, those we usually find in the 70% to 80% range in favor of having abortion legal in most of or all cases. We have been doing a really deep dive into the topic of abortion over the year of 2022. So in a couple of weeks we're going to release a large report that has data for the nation as well as all 50 states on abortion legality, views of the overturn of Roe, and how much abortion impacts people's vote choice. Does the candidate have to share their views on abortion to get their vote? So in addition to having all 50 states, we have a really deep dive on all the religious groups that we track, and with this huge study, we are able to go pretty deep. As far as over the past year, opinions on legality really haven't shifted a lot. The same religious groups are still in favor and opposed. Some of the more interesting places are when you look at the middle groups. They're not super in favor, but they're not white Christian evangelical opposed, either. White mainline Protestants, Catholics, Black Protestants, Hispanic Catholics, folks in those categories I think tend to favor legal abortion a bit more than people would expect. I think especially the Catholic piece, given the official church stance, or the back and forth in the church stance. In general, the public among both white and Hispanic Catholics and Catholics of any other race or ethnicity, they're generally majority in favor of legal abortion. I mean, we're a country that is fundamentally in favor of legal abortion in most or all cases except for white Evangelicals and Republicans. David Beard: And I think one of the things that I'm sure you all find when you do these studies is that folks' beliefs do not line up perfectly with their religious identifications, most obviously with the Catholic Church because they do have such set out dogma from Rome, but even into other denominations or other religions. Just because a religion has certain beliefs doesn't mean that the American population who adheres to that religion is 100% signed up for those beliefs. Natalie Jackson: Right, absolutely. And one of the other things that we see in the data is that a lot of these people that claim affiliations may or may not be active. They may not actually attend services or be active in a religious community. The exception to that is the evangelicals again. The evangelicals are typically much more engaged, but outside of that group, you may or may not be even very attached to your religion at all. David Beard: So you mentioned rigorous high quality polling as the type of polling that you all do. I want to take you back to last year and all of the election polling that took place. And there was a lot of debate after Democrats I think did better than expected on election night. There were a lot of hot takes. Some people were like, "The polls were bad." Some people were like, "Well, some of the polls were bad." And some people were like, "Actually, the polls were good." So I just wanted to get your broad view on how you think the election polling industry did in 2022. Natalie Jackson: This is a really good meaty topic because it's right at the intersection of how communication and media mix in with the art and science of polling. And what we saw happen in October and November, both right before and right after the election, was a big crash of those two things not necessarily matching up. It's true that there was a wave of polls in September, October that showed some Republican candidates doing better. There were some unbelievable ones. For example, Washington Senate where we had some polls showing close races that really should not have been close. And as it turned out, they weren't close. We also had a lot of polls showing things as they were, showing close races where the races were indeed very close that were showing Republicans would slightly win the generic congressional ballot. Republicans did slightly win the generic congressional ballot. The main issue that I see that happened pre-election was the tug of war between what people thought would happen and what they assumed would happen based on what was supposed to happen in a midterm and not wanting to get overly hopeful. The polls missed in 2020 and 2016. They undercounted Republicans. Are they under counting Republicans again? The natural instinct for everyone was to err on the side of assuming that Democrats were being overestimated and that Republicans would do better. When you combine that with the fact that you had this wave of polls showing unexpectedly good results for Republicans, you had a perfect storm for misinterpreting the reality of the data in total and the reality of what was going on. My take is there were a few not great pollsters. There always are. That always happens. The low cost of data collection and the ability to disperse it on the internet and the ability to get publicity by getting your poll listed on FiveThirtyEight, all of these things incentivize not-super-serious people to jump in and throw out an election poll for whatever agenda they have in mind, even if that's just getting their name out there. Then, after the election, everybody jumped all over the polls that had said there would be a Republican surge or indicated that there might be a Republican surge, and said, "Oh my gosh, the polls were wrong," when in reality that was a small number of polls compared to the full picture. But those small number were amplified by the pundits and the idea that what was supposed to happen in a Midterm was the President's party would lose big. If you look at the full picture, if you look at all the polls, there were a lot of polls that were really, really accurate. They just weren't the ones that people were talking about. We need to pay attention to the pollsters that have been around a long time. They do careful work. Those people, by and large, they may have had some misses in 2016 and 2020, but they've done a good job of pivoting, adjusting, getting things back into place, and you saw those pollsters do quite well in 2022. David Beard: So I think 2022 was really one of the best examples of a year where the good polls hit it right and a lot of these bad polls that are very cheaply made and a lot of them tended to skew Republicans really missed, and sometimes missed really badly. But I think looking before the election, the question that our listeners might want to know is, what is a good poll? How can they know if a poll is well conducted and worth paying attention to, or something that you should really look at with some suspicion beyond, obviously, if it's a partisan poll, everybody knows that's maybe a factor, but beyond that initial look? Natalie Jackson: Overall, it's really gotten tricky to say what a "good" poll is versus a bad poll in part because we used to be able to say, "Well, polls conducted this way using a live telephone caller methodology, calling landlines and cell phones, this is the best quality poll." And you'll still hear that view, but there's really no support for it anymore in the data. There are multiple ways of doing polls that are equally good. The challenges of getting people to answer their phones has rendered telephone polling a lot trickier. It's harder to get a representative sample and balance it correctly and all of that, when nobody answers their phones. So where we used to have some shortcuts that we could tell people this is the gold standard, that really doesn't exist anymore. So what I tell people to look for instead is a methodology. So most pollsters at the end of a release, or somewhere in it, will have a methodology statement. And the things you want to know are how did they talk to people, how many people did they talk to, do they give you a margin of error, some estimate of the uncertainty associated with the poll, and does it sound like all of that is a reasonable way to get in touch with a representative sample of Americans? And this is where I kind of tell people to rely on their common sense. Right? If your Spidey-senses jump up and say, that doesn't sound like a really good way to do this, go look at a different poll. If you can't find that methodology at all, if they're not telling you how they did it, if they're saying things like, "It's proprietary," there's no reason you should pay attention to that poll because there are going to be plenty of other pollsters out there, particularly in the elections game, there are going to be plenty of others who do tell you how they do their polls that you can go look at instead. You might have to wait a few days. It doesn't hurt. I know everybody wants to know things all the time right away, but it won't kill you to wait a couple days for a better quality poll to come out. During an election, it's a matter of days there's going to be a good poll. As far as online polling, there's a lot of different ways to poll online. So that's where I tell people, read how they did it. If you don't understand what they're saying or you don't think it seems like enough information or something that you can get, go to a different poll. David Beard: Sure. I mean, asking people on the internet to wait a couple days for something is a tall ask, but I think our listeners can do it. So you mentioned FiveThirtyEight earlier. That's obviously sort of the behemoth of the election model forecasts. You now see a lot of them if you're in sort of this online election world. RealClearPolitics is obviously a well-known polling aggregator. They don't really have a model, but they aggregate polls. They have a lot of controversy around them for which polls they include and various questions around them. So how should folks evaluate these election models? FiveThirtyEight. Are they useful? We obviously saw a lot of these bad polls included, so what should we do with them? Should we throw them out or should we try to find a better way, keep reading them? Natalie Jackson: The entire aggregate philosophy is to make life easier for people. The idea is instead of looking at a whole bunch of polls, you look at one thing, you get one number, you're like, okay, that's where it is. Great. I don't have to chase down all these different polls and figure out how they were done and what I believe and all of that sort of thing. The direction that it's gone in the last little bit, probably eight-ish years, where we have RCP and FiveThirtyEight as the main players in the space, is that you can either look at a very simple average, which is RCP. They've never been very forthcoming about what their criteria are for including polls. For what it's worth, though, their aggregate for generic congress ballot in 2022 was pretty right on and not that far off of FiveThirtyEight. So for all the kind of talk there, it actually looked pretty similar on that big picture piece. With FiveThirtyEight, you have to understand that when you look at their aggregate, you're taking on all of the assumptions that they've written into that model. And there are a lot. So they adjust polls by house effects. So if the poll is more skewed toward Democrats than most other polls are, they'll kind of give it less weight. If it's skewed more toward Republicans, maybe it gets a little less weight. They also, when they forecast, they do an extension of the aggregate and they weight based on their pollster grades. They give pollsters grades based on their accuracy, which is a bit of a problem because you're judging pollsters based on their past performance, which for longtime pollsters might be pretty good, but for short timers that have just come into the field in the last couple of cycles, you don't really know what their reputation is yet. So that can be a little bit tricky. So you're taking on all of the assumptions in FiveThirtyEight's model when you look at theirs. And I think there's not a lot of understanding about that. Instead of being selective in the polls, they take everything and then build these assumptions into the model to make adjustments. And I think that is well over most people's heads. No offense to anyone listening, I'm sure you're quite statistically savvy, but- David Beard: I don't understand that. I will happily admit that when I try to read the details of the model, I totally get lost. Natalie Jackson: Yeah. So it's just a more advanced picture and you have to believe, just kind of put blind faith that they're doing it right. And for what it's worth, I do think all of the things they put in are generally fairly reasonable. It's just a level of complexity that I think is often more confusing than it is illuminating. David Beard: So as you mentioned with these two sort of at different ends with RCP being this aggregator that's very weirdly selective with their polls, but very simple and then FiveThirtyEight that takes everything and makes this very, very complicated model. Is there space for something in between that sort of tries to aggregate good polls, tries to make a clear determination ahead of time of what polls will be included and sort of shows this aggregation in a very straightforward manner? And if so, why hasn't someone done it? Natalie Jackson: There's absolutely a space and a need for that middle ground, and if someone wants to do it, they should get in touch with me. So I think there is an opening here. FiveThirtyEight is going to continue doing what they're doing. RealClearPolitics is going to continue doing what they're doing. A middle ground, which is what we tried to do at HuffPost Pollster was... It was great. We had a lot of nice features on the page. You could actually select which types of polls you wanted included and it would run, not a full model because that took too much time, but it would run a quick little regression line and you could get a rough estimate based on only the types of polls that you wanted to see. And we liked that model. We really enjoyed creating that, sharing it with people. It seemed to be a good space where we were clear about what's included, what's not. We had a set of standards and we weren't putting in a lot of special sauce into our modeling. We were just making it a straightforward model with appropriate estimates of uncertainty around what we were saying. David Beard: And I assume that a big factor, obviously is cost because these types of models that do everything with the polling get a lot of attention, but they get a lot of attention for a few months every two years. And then the rest of the time you have to figure out if somebody's paying a staff or paying for a website, obviously you need money to do that, where maybe there's not a lot of attention in February of 2023. Natalie Jackson: Right. Yeah, I mean that is one of the issues. The other issue is these things tend to be in the media space and the media space is constantly trying to figure out where its best positioning is. We've seen a lot of back and forth with media doing layoffs and then pivoting and changing directions. And with the media industry itself being in flux trying to adapt between over the last 20 years where the industry has just fundamentally changed business models in a lot of ways, there's not as much space to make a pitch for give me a really, really good web developer, several people running down polls every day and somebody to code up models and we can make this great thing. That said, the Pollster chart's got millions of views, so in the right space with the right business model, it could be quite lucrative. And again, I know how to build those models. David Beard: I was a big fan back when Pollster was live. I was definitely a lot of those views. So hopefully, something like it will return someday. Now, before we let you go, I did see that you had just finished and submitted a manuscript for a book. Is there anything you can preview for us about what that might be about? Natalie Jackson: Sure. It's a little bit off on a different topic. So one of my side passion projects is helping people like me understand as they're going through graduate school, what their career options are. Because there's a lot of bias, when you're in a PhD program, that you have to go into academia. Well, academia doesn't have as many jobs as there are PhDs produced. And there's, up until really the last few years, there's been a lot of lack of understanding of what the career pathways are for people who are really academic-y but don't want to necessarily work in a university. So this book is an edited volume of different career pathways. I've got a bunch of different chapters on different directions you can go with a social science PhD. So it's pretty wonky and kind of off on a side trail, but I've really viewed it as kind of a service project to people who are in a similar pathway coming up behind me just to try to help them out. David Beard: Absolutely. It sounds like a great resource for people in that situation. And then finally, where can people follow you and follow your work going forward? Natalie Jackson: I am on Twitter and Mastodon as Natalie MJ10, and PRII is prri.org. You can see all of our reports there. Lots of interesting things. And of course you can look me up on LinkedIn as well. David Beard: We've been speaking with Natalie Jackson, the director of research at PRRI. Natalie, thank you again for joining us. Natalie Jackson: Thank you. It's been great. David Beard: That's all from us this week. Thanks to Natalie Jackson for joining us. The Downballot comes out every Thursday everywhere you listen to podcasts. You can reach out to us by emailing thedownballot@dailykos.com. If you haven't already, please subscribe to the Downballot on Apple Podcast and leave us a five star rating and review. Thanks to our producer, Cara Zelaya, and editor Trever Jones. We'll be back next week with a new episode. [END] --- [1] Url: https://www.dailykos.com/stories/2023/2/9/2151934/-The-Downballot-When-is-a-poll-a-quality-poll-with-Natalie-Jackson-transcript Published and (C) by Daily Kos Content appears here under this condition or license: Site content may be used for any purpose without permission unless otherwise specified. via Magical.Fish Gopher News Feeds: gopher://magical.fish/1/feeds/news/dailykos/