Myths & Delusions Of IHP’s ‘Sri Lanka Opinion Tracker Survey’

By Ananda Jayawickrama –

The Institute for Health Policy (IHP in short), as described on its web page, is a Colombo based independent research institution working to improve health and social systems in Sri Lanka through quality research analysis.  Though it is not within the direct mandate of the institute, IHP is conducting monthly opinion surveys, titled Sri Lanka Opinion Tracker Survey (SLOTS), on presidential election voter intention. The survey reports time series results on monthly basis since Oct 2021, but survey press releases are available on IHP website only for the months of Aug, Nov and Dec in 2022 and Sep, Oct, Nov and Dec in 2023. So, it is not clear whether IHP have conducted the survey regularly on monthly basis or whether IHP have collected data for some months through ‘past tracking method’. The main objective of the SLOTS is to estimate and assess popularity of potential presidential candidates, namely Sajith Premadasa (SP), Anura Kumara Dissanayake (AKD), Ranil Wickrdmesinghe (RW) and a candidate of Sri Lanka Podujana Peramuna (SLPP).

Sample and data

According to IHP, the SLOTS uses interviews from a ‘national’ sample of adults (age 18 and above). Interviews have been conducted through telephone (mobile phone) calls selected by random digit dialing of mobile numbers. These telephone call interviews are merged with ‘a physical interview sample coming from a national panel of respondents with random selection’ to form a monthly sample of 553.37 persons on average. The SLOTS report also states that monthly sample information is used with historical sum (cumulative) sample to readjust opinion of voters on the presidential candidates.

In 2022, there were 16,692,398 (16.692398 mn) voters in the country according to the Election Commission. Thus, an average sample of 553 voters is 0.0033% of total voters or simply 3.3 voters per 100,000 voters. Since the sample is non-representative of the national census, outcome of such small sample should be read cautiously. In an opinion survey, what is most important is having a representative sample covering all geographic and demographic diversities/ factors and their relative size. How people opinion on a presidential candidate in deep South and far North should be considered. The SLOTS report is silent on what weights are given to diverse factors of vote base of the country, such as geographical location, ethnicity, religion, gender, age, education, and so on. The SLOTS also does not present a discussion on the possibility of a sample unit be selected repeatedly in different months. If sample units be selected repeatedly, the accumulation of sample units over months to form a bigger sample is fundamentally incorrect: if this happens it is equivalent to a situation where a voter enjoys multiple voting rights.

If monthly samples are mutually exclusive, re-polling is needed in the cumulative historical sample. It seems to me that IHP is using monthly sample as the basic decision-making sample and the cumulative sample as the census/ national sample used for final adjustments. If repolling is not done in cumulative sample each month and decisions are made adjusting parameters of the current month sample to the cumulative sample, IHP is doing a fundamental and grave mistake in statistical analysis and a crime to the democratic voting rights.

Analytical methodology

As stated in the SLOTS reports, IHP estimates voting intent using Multilevel Regression and Post-stratification (MRP) model and exploits data from all SLOTS interviews to estimate voting intent in a particular month. In general, the MRP model is widely used in two closely related applications: (i) using national surveys such as census to predict sub-national level or small area opinion, (ii) using non-representative surveys. We presume that IHP is using MRP under the second application where their sample is non-representative.

The MRP estimation process follows two important stages: building individual response model and using individual model parameters for post-stratification.  First, individual responses to a survey, probably to a national survey are modeled in multilevel logistic modeling framework in order to predict opinion estimates based on demographic-geographic subgroups/diversities (e.g. voting intent of a middle-aged Sinhalese female with secondary education resided in Anuradapura district). Secondly, these opinion estimates of a subgroup obtained based on sample survey are weighted by the frequency of the subgroup at the national survey of interest, may be census of population or such national wide survey. This second stage is known as the post-stratification which is used to re-stratify sample estimates based on census information. For better stratified estimates the factors and their values in the sample should match with the corresponding factors and their values in national survey/census.  Thus, the MRP methodology is limited and restricted to the analysis of factors that are present in both survey sample and the national survey. For example, a question on voting intent can be analyzed by the MRP method if that question includes in survey sample and the national survey. Further, there should be a matching between the values of factors in both survey sample and the national census. Therefore, the MRP process requires the verification of similarity of data coming from survey sample and national survey/census and also a series of tests for model accuracy. Once the model clears these tests it can be used for the analysis. The MRP method of estimation of opinion polling is a complex procedure and it needs sound data to reach at sensible results.

Though IHP states that it follows MRP procedure of estimation, it is in question as related details are not revealed. Since the SLOTS sample is non-representative the task of continuous impunity correction is more difficult and the report is silent about how it has been done. We presume that IHP considers its monthly SLOTS sample as the survey sample and the accumulated monthly survey sample (cumulative survey sample) as the national level survey or census. If that is the case, the results of IHP’s SLOTS analysis is blatantly wrong and its predictions are not valid.

General procedural and analytical issues

The IHP states that it conducted SLOTS interviews from Oct, 2021 and Jan 2024 for 27 months, with 14,941 total interviews. The number of average monthly interviews is 553.37. If this average number is accumulated for 27 months, it would amount to 14,941 total interviews. Interestingly, SLOTS preference shares of SP, AKD, RW and SLPP candidate sum exactly to 100 in each month. That means there are no any other candidates, undecided votes, non-responses, rejection to give answers, etc. which are common in a sample survey and especially in an opinion survey. The opinion on presidential election given by a person in Oct 2021 can be different in Oct 2022, Oct 2023 or any other month for that matter. It is not clear whether the IHP allows for such dynamics in their analysis.

The recent SLOTS report states that “All estimates are adjusted to ensure that the sample matches the national population with respect to age, sex, ethnicity, socioeconomic status, education, geographical location and voting in 2019 presidential election and 2020 general election”. It is not clear how sample estimates have been adjusted to ensure sample matches with national characteristics. The above statement seems to be incorrect for many reasons: (i) sample (553.37 persons on average) is too small to match all characteristics of the census; (ii) adjustment can be made only for variables and factors which are present in both survey sample and national sample/census. As stated in the SLOTS reports, data on 2019 presidential election and 2020 general election cannot be matched with SLOTS sample information as IHP does not have details of voters such as age, gender, education, ethnicity, religion, socio-economic status, etc. of the above two elections.

One of the issues needs serious concern is that given fluctuations in preferences (see SLOTS time series graphs and Table 1 above) require ups and downs in vote turn-out of a particular candidate. Though increases in vote turn-out is possible with a new monthly sample, negative vote turn-out is impossible even in the cumulative sample if the repolling is not done. The lowest possible number of votes a candidate can receive in a given monthly sample is 0. But this is not the case with SLOTS. For example, in Nov 2023, SP gets 454 additional votes, AKD gets 427 additional votes, RW loses 460 votes and SLPP candidate losses 100 votes. This requires 321 voters in cumulative sample to change their preference. In Dec 2023, SP gets 327 additional votes, SLPP candidate gets 188 additional votes, AKD loses 12 votes and RW loses 94 votes. This requires 153 voters in SLOTS cumulative sample to relinquish their preferences in Dec 2023. But this behavior has never been examined and cannot be examined in a random digit dialing framework. We further analyze information given in Table 1 under two scenarios.

Scenario 1

In this case we use the cumulative sample (see Table 1) for the analysis though using the cumulative sample as done in SLOTS is incorrect. As per SLOTS reports, the popularity of AKD increases from 10% in Oct 2021 to 50% in Dec 2023, reaching the highest 52% in Nov. 2023. Then, it seems that IHP applies end preference rate (prevail in Dec 2023) to the cumulative sample to receive 7470 supporters for AKD out of 14,941 individuals (50%). This computation increases 55 AKD supporters from 553 individuals (10%) in Oct 2021 to 7470 supporters from 14,941 individuals (50%) in Dec. 2023. This means that vote turn-out of AKD has increased by 13,482% between Oct 2021 and Dec 2023 (or by a monthly average rate of 500%). It also indicates a 400% increase in AKD’s vote share, an unbelievable and amazing popularity gain, during the period. If this rate is applied to the number of votes obtained by AKD in 2019 presidential election (418,553 votes), the number of votes that AKD would have received in Dec. 2023 exceeds 56 mn. But the country’s vote base is limited to 17 mn by the end of 2023. The same procedure with the number of votes SP received in 2019 presidential election (5,564,259) would predict a massive unrealistic vote base for SP. These unrealistic estimates testify the fallacy of the procedure adopted by the IHP in its SLOTS analysis. The cumulative sample may generate amplifying effects as there is no mechanism to counter the issue of counting many times and the application of current preference rates to previous samples.

Scenario 2

In scenario 2, we use only monthly sample of IHP even though there are serious issues with their monthly sample as well. AKD received 55 votes out of 553 interviews conducted (10%) in Oct 2021 and 277 votes out of 553 persons (50%) in Dec 2023 on average. In this case, AKDs vote base has increased by 404% in between Oct 2021 and Dec 2023.  If this growth rate is applied to the number of votes obtained by AKD in 2019 presidential election, the number of votes that AKD received in Dec 2023 would be 2,109,507 (2.11 mn), about 15.4% of the estimated total valid votes (80% of registered votes in 2023, see Table 2 also). This share of votes is a realistic and achievable number for NPP if it continues to keep its popularity (50%) as given in SLOTS samples.

On the other hand, according to SLOTS, SP received 172 votes out of 553 interviews conducted (31%) in Oct 2021 and 182 votes out of 553 persons (33%) interviewed in Dec 2023. In this case, SPs vote base has increased by 5.81% in between Oct 2021 and Dec 2023.  If this growth rate is applied to the number of votes obtained by SP in 2019, the number of votes that SP received in Dec 2023 would be 5,887,762 (5.89 mn), about 43.2% of the estimated total valid votes in 2023. It should be noted here that SP requires only 38.5% of popularity in the IHP 553 persons sample with 2019 vote share to cross 50% bar of presidential election. It seems, knowing this possibility, IHP is attempting to restrict the popularity rate of SP at about lower 30s since July 2023.

Biasness in reporting

Opinion poll researchers and institutes are generally value neutral and unbiased as they are independent or pretended to be independent. The neutrality and independency should be maintained for the faith of all segments of the population on survey outcomes. The survey institute should not pass value judgements based on the opinion survey that would harm or benefit any group in concern. It seems that these research ethics have been violated by the IHP SLOTS researchers. Consider the following statement of the Research Director of the survey  “…, it does appear that AKD would have likely won a presidential election in October (2023) even against a joint SJB UNP ticket” (see page 2 of Nov 2023 SLOTS press release). The Director needs to explain how he has arrived at this conclusion. It is important know that why he ignored the fact that a joint SJB-UNP is different from SJB and UNP for the opinion of people. Also consider the following statement given in IHP Nov 2023 press release “Establishment parties wanting to compete with NPP probably need to be doing some serious soul-searching”. This clearly indicates biasness of the IHP towards a candidate. Since survey does not reveal any reason for the voting intent, asking other parties for ‘soul-searching’ indicates that the course of action of one party is considered superior to other parties. These statements by the IHP clearly violates research ethics and shows the degree of hypocrisy and partiality associated with the institute.

Ground reality

Though IHP has already selected their next president of Sri Lanka through a miserable and poor sample analysis, the ground reality is different. It is of course obvious that AKD and his NPP is currently getting an improved support especially in some parts of the country compared to their previous attempts. A segment of the frustrated GR supporters looks for AKD and NPP as their survivors. The question is whether that improved popularity is good enough for them to cross 50% requirement of valid votes. It led me to look at the potentiality of presidential candidates, SP and AKD, based on their 2019 presidential election results.

With a 2.07% annual average growth rate of registered votes (between 2020-2022), we estimate the country’s registered votes in 2023 to be 17.04 mn and valid votes in 2024 presidential election to be 13.63 mn assuming a 20% uncast and rejected votes. These numbers indicate that a candidate should obtain minimum 6.82 mn votes to cross the bar. Assuming their 2019 presidential votes remain intact, Sajith Premadasa requires additional 1.3 mn votes, about 18% of the minimum requirement, and Anura Kumara Dissanayake requires 6.4 mn additional votes, about 94% of the minimum requirement. It seems me that Anura Kumara Dissanayake has an uphill task ahead compared to Sajith Premadasa, since NPP leader is required to entice more than six million voters who voted for GR in 2019 against him. NPP leader needs to increase his vote turn-out by 1528% or more than 15 times compared to his previous attempt. In the mean-time, SJB leader Sajith Premadasa requires an increase in his vote turn-out by 23% or 1.15 times to reach the target. These numbers tell us that each NPPer, voted for AKD in 2019, needs to breed 15 additional AKD supporters and every seven SP supporters, voted for SP in 2019, need to breed one new SP supporter for their leader to cross the bar of 2024 presidential election. The easiest task of the two will decide the next president of the country.

The post Myths & Delusions Of IHP’s ‘Sri Lanka Opinion Tracker Survey’ appeared first on Colombo Telegraph.