07-Estimating conflict losses and reporting biases

PNASBRIEF REPORTPOLITICAL SCIENCES<img src="/media/202408//1724856292.900314.png" />OPEN ACCESS<a href="http://crossmark.crossref.org/dialog/?doi=10.1073/pnas.2307372120&amp;domain=pdf&amp;date_stamp=2023-08-09"><img src="/media/202408//1724856292.967633.png" /></a>Estimating conﬂict losses and reporting biases<img src="/media/202408//1724856292.998296.png" />Benjamin J. Radforda,b,1 <a href="https://orcid.org/0000-0002-8440-0655">ID ,</a> Yaoyao Daic <a href="https://orcid.org/0000-0003-3035-331X">ID ,</a> Niklas Stoehrd, Aaron Scheine, Mya Fernandezb,c, and Hanif Sajida <a href="https://orcid.org/0009-0007-6769-9484">ID</a>Edited by David Laitin, Stanford University, Stanford, CA; received May 2, 2023; accepted July 17, 2023Determining the number of casualties and fatalities suffered in militarized conﬂicts is important for conﬂict measurement, forecasting, and accountability. However, given the nature of conﬂict, reliable statistics on casualties are rare. Countries or political actors involved in conﬂicts have incentives to hide or manipulate these numbers, while third parties might not have access to reliable information. For example, in the ongoing militarized conﬂict between Russia and Ukraine, estimates of the magnitude of losses vary wildly, sometimes across orders of magnitude. In this paper, we offer an approach for measuring casualties and fatalities given multiple reporting sources and, at the sametime, accounting for the biases of those sources. We construct a dataset of 4,609 reports of military and civilian losses by both sides. We then develop a statistical model to better estimate losses for both sides given these reports. Our model accounts for different kinds of reporting biases, structural correlations between loss types, and integrates loss reports at different temporal scales. Our daily and cumulative estimates provide evidence that Russia has lost more personnel than has Ukraine and also likely suffers from a higher fatality to casualty ratio. We ﬁnd that both sides likely overestimate the personnel losses suffered by their opponent and that Russian sources underestimate their own losses of personnel.news bias j war j casualties j open-source data j Bayesian statisticsIn February 2022, Russian armed forces invaded Ukraine, expanding upon their previous annexation of Crimea and eastern parts of the country in 2014. Since that time, governments, NGOs, and open-source investigators (OSI) have produced thousands of estimates related to physical losses suffered by belligerents in the conﬂict: casualties, fatalities, and equipment losses. These reported losses form an incomplete time series that provides snapshots of the conﬂict up until the point at which the claim is made. However, these reported losses are also obscured by the fog of war and often contradict one another. Contemporaneous reported numbers of cumulative incurred losses made by different sources may differ by orders of magnitude. For example, on September 21, 2022, Russian Defense Minister Shoigu reported that 5,937 Russian soldiers had been killed in the conﬂict (1). However, during the same week, the Ukrainian Ministry of Defense reported that 55,510 Russian soldiers had been killed (2).Downloaded from https://www.pnas.org by 5 1. 195.24 1.227 on December 3, 2023 from IP address <a href="51.195.241.227">51.195.241.227</a>.We construct a dataset of reports of losses suffered by Russia and Ukraine to predict the daily losses per side and per loss category, where categories include various types of equipment as well as personnel. Furthermore, we account for correlations between loss categories to adjust for gaps in reporting while also accounting for source-speciﬁc biases in the original reporting. Under this model, we can predict the expected losses suffered by both sides of the conﬂict for every loss category at the daily and cumulative levels. We ﬁnd evidence that Russia has lost substantially more personnel than has Ukraine and also likely suffers from a higher fatality to casualty ratio. However, relative equipment losses tend to be closer to parity between sides. We also ﬁnd that Russian sources overestimate Ukrainian personnel losses while underestimating their own.Measuring the casualty and fatality rates of a military conﬂict is important both for characterizing the conﬂict and for forecasting its progression. Many deﬁnitions of war, for instance, depend on knowledge of both the absolute and relative number of combatant fatalities among belligerents (3). Fatalities themselves are sometimes used as a (near-)continuously valued proxy for concepts that are difﬁcult to measure, like conﬂict severity (4) or conﬂict escalation (5).Assessments of fatality and casualty rates and the number and types of equipment available to the opposing side are crucial for war planning, managing public opinion, and the protection of human rights. There is a long history of statistical modeling in the service of estimating the costs of war. In World War 2, statisticians used consecutive serial numbers on captured German tanks to estimate monthly tank production capacity (“the German tank problem”) with greater accuracy than intelligence analysts (6). At home, political leaders have an interest not only in understandingPNAS 2023 Vol. 120 No. 34 e2307372120Author aﬀiliations: a Public Policy Program, University of North Carolina at Charlotte, Charlotte, NC 28223; b Intelligence Community Center of Academic Excellence, Department of Political Science &amp; Public Administration, University of North Carolina at Charlotte, Charlotte, NC 28223; c Department of Political Science &amp; Public Administration, University of North Carolina at Charlotte, Charlotte, NC 28223;d Department of Computer Science, ETH Zürich, Zürich 8092, Switzerland; and e Department of Statistics, University of Chicago, Chicago, IL 60637Author contributions: B.J.R. designed research; B.J.R., M.F., and H.S. performed research; B.J.R., Y.D., N.S., andA.S. analyzed data; and B.J.R., Y.D., and H.S. wrote the paper.The authors declare no competing interest.Copyright © 2023 the Author(s). Published by PNAS. This open access article is distributed under <a href="https://creativecommons.org/licenses/by/4.0/">Creative</a> <a href="https://creativecommons.org/licenses/by/4.0/">Commons Attribution License 4.0 (CC BY).</a>1To whom correspondence may be addressed. Email:<a href="mailto:benjamin.radford@charlotte.edu">benjamin.radford@charlotte.edu.</a> Published August 14, 2023.<a href="https://doi.org/10.1073/pnas.2307372120">https://doi.org/10.1073/pnas.2307372120</a> 1 of 31e+0510000Tank Losses Mi litary Deaths1000100100100001000100100Russia Ukraine<img src="/media/202408//1724856293.163508.jpeg" /><table><tr><td></td></tr></table><img src="/media/202408//1724856293.226698.jpeg" /><table><tr><td><img src="/media/202408//1724856293.3294442.png" /></td></tr></table><img src="/media/202408//1724856293.512167.png" /><table><tr><td><img src="/media/202408//1724856293.545579.png" /></td></tr></table><table><tr><td rowspan="2"></td><td></td></tr><tr><td><img src="/media/202408//1724856293.576436.png" /> GB Source o OSI△ Other<img src="/media/202408//1724856293.632811.png" /> RU Source X UA Source◇ United Nations <img src="/media/202408//1724856293.745693.png" /> US Source<img src="/media/202408//1724856293.811656.png" /> Daily (all sources)</td></tr></table>0 100 200 300 0 100 200 300Days since Feb. 24. 2022 Days since Feb. 24. 2022Fig. 1. Estimated military personnel (Top) and tank (Bot- tom) losses incurred by Rus- sia (Left) and Ukraine (Right) during the first 365 d of war. Expected daily losses are de- picted with a solid line, and expected cumulative losses are depicted with a dashed line. Note the 95% poste- rior credible intervals in gray. Points indicate reports from the sources given in the leg- end in the Lower right. Y-axis values are sinh-1 (y) trans- formed for clarity.losses but in managing the public’s understanding of losses. In the context of the United States, the public conditioned its support for military action against Iraq, at least in part, on perceptions of the expected casualty numbers (7). The importance of public perceptions of losses during a military conﬂict is underscored by media reports from April 2023: Leaked US intelligence documents apparently revealed internal estimates of Ukrainian and Russian losses during the ongoing conﬂict. However, reports also indicate that the numbers in the documents were likely modiﬁed by a third party at some point to minimize Russia’s losses (8). Recent work also suggests that governments will underreport violence against noncombatants (9). This highlights the importance of accounting for the biases of speciﬁc sources when estimating losses during military conﬂicts.Downloaded from https://www.pnas.org by 5 1. 195.24 1.227 on December 3, 2023 from IP address <a href="51.195.241.227">51.195.241.227</a>.Unfortunately, as Clausewitz writes, “casualty reports on either side are never accurate, seldom truthful, and in most cases deliberately falsiﬁed” (10). Accurate casualty estimates are “notoriously difﬁcult” (11). Methods for estimating war deaths can be grouped into two primary categories: Those that rely on contemporaneous reports, often from multiple sources, and those that rely on postwar surveys or demographic studies (12). One notable effort to catalog combat-related deaths is the PRIO Battledeaths Dataset (13). This effort, like ours, falls primarily within the former estimation methodology. Unlike our approach, the PRIO Battledeaths Dataset reports only annual battle-related deaths per conﬂict per country. In contrast, we attempt to leverage the temporal aspect of casualty reports to estimate losses at the daily level. We also leverage reports of multiple loss categories to help ﬁll gaps in reporting for any single loss category, under the assumption that losses in some categories will be correlated with others. For example, losses of armored vehicles are likely correlated with casualties (14).Our data contain 4,609 claims of losses reported on social media, news venues, and by various government sources. We aggregate these sources into seven groups that we refer to as “claim sources”: OSI (n = 169), Russian sources (247), UK sources (32), Ukrainian sources (3,858), the United Nations (78), US sources (71), and other sources (154).* Losses are recorded at two levels, daily and cumulative, with the latter comprising 96.5% of all observations. We record losses for 21 categories, 14 of which are given in Table 1 and include military and civilian fatalities, casualties, and losses of various types of equipment. Due to * Source refers to the originator of a claim (e.g., US Department of Defense or Russian Ministry of Defense) and is distinct from the venue that reported on a claim (e.g., CNN or NYT).2 of 3 <a href="https://doi.org/10.1073/pnas.2307372120">https://doi.org/10.1073/pnas.2307372120</a>reporting inconsistencies, all time series are incomplete and many contain inconsistent observations: nonmonotonically increasing cumulative values, missing values, and multiple contradictory values on a single day. We use all available data to estimate daily and cumulative losses for all loss categories while accounting for claim source–speciﬁc biases in the reports.ResultsWith our model, we estimate expected daily and cumulative losses for every loss category and target country pair (“category– target”), conditioned on estimated claim source biases. Fig. 1 depicts the data and posterior predictions for military personnel deaths and tank losses for both major parties to the conﬂict. We ﬁnd that Russian personnel losses have outpaced Ukrainian personnel losses, with expected loss numbers of 76,687 (95% credible interval: 38,670–139,772) and 17,223 (6,219–39,105), respectively, as of February 23, 2023. We compute the ratio of casualties to deaths for Russia and Ukraine, ﬁnding values of 2.9:1 and 4.9:1, respectively.In the lower two panels of Fig. 1, we see expected losses of tanks over time. Both Russia and Ukraine are estimated to have suffered similarly here, with expected losses of 3,380 and 2,051, respectively. The uncertainty, indicated visually by the gray-shaded 95% posterior credible intervals, is much higher for Ukraine,though, obtaining lower and upper bounds of 385 and 5,946 by the end of the ﬁrst year of war, versus Russia’s bounds of 1,704 and 6,178.Table 1 presents a selection of estimated cumulative losses as of February 23, 2023, alongside the number of reports corresponding to each loss type (n), and the bounds of a 95% posterior credible interval for each estimate.We also estimate the biases exhibited by claim sources with respect to loss categories. Bias does not necessarily imply intentional misrepresentation but rather any systematic over- or underestimation relative to our estimated loss values. When looking at Russian military deaths, we ﬁnd that, for every loss suffered, Russian sources report only 0.3 losses (0.1–0.5). This roughly corresponds to the Russian account of 5,937 losses by September 21, 2022, at which point our model estimates Russia had likely lost 31,532 soldiers. Russian sources overestimate Ukrainian military deaths at a rate of 4.3 to 1. Ukrainian sources overestimate Russian deaths by nearly double, though no bias is supported in the 95% CI (1.0:1–3.4:1). We ﬁnd no evidence of systematic bias in Ukrainian reports of Ukrainian military deaths.pnas.orgTable 1. Estimated cumulative losses as of February<a id="bookmark1"></a>23, 2023ISO2 Category n Est. 95% CI<table><tr><td>RU</td><td>AA Systems</td><td>233</td><td>339</td><td>[76–1,070]</td></tr><tr><td>UA</td><td>AA Systems</td><td>13</td><td>1,105</td><td>[108–5,247]</td></tr><tr><td>RU</td><td>Armored Vehicles</td><td>400</td><td>6,351</td><td>[2,966–11,791]</td></tr><tr><td>UA</td><td>Armored Vehicles</td><td>15</td><td>3,280</td><td>[777–8,439]</td></tr><tr><td>RU</td><td>Artillery</td><td>380</td><td>1,483</td><td>[701–2,818]</td></tr><tr><td>UA</td><td>Artillery</td><td>35</td><td>2,290</td><td>[519–6,966]</td></tr><tr><td>UA</td><td>Civilian Casualties</td><td>21</td><td>38,155</td><td>[13,245–84,852]</td></tr><tr><td>UA</td><td>Civilian Deaths</td><td>46</td><td>13,287</td><td>[4,081–32,399]</td></tr><tr><td>UA</td><td>Civilian Injuries</td><td>26</td><td>19,464</td><td>[5,396–46,460]</td></tr><tr><td>RU</td><td>Helicopters</td><td>389</td><td>172</td><td>[87–311]</td></tr><tr><td>UA</td><td>Helicopters</td><td>30</td><td>64</td><td>[14–183]</td></tr><tr><td>RU</td><td>Jets</td><td>409</td><td>146</td><td>[68–273]</td></tr><tr><td>UA</td><td>Jets</td><td>38</td><td>122</td><td>[32–372]</td></tr><tr><td>RU</td><td>Military Casualties</td><td>130</td><td>218,800</td><td>[108,432–397,361]</td></tr><tr><td>UA</td><td>Military Casualties</td><td>16</td><td>75,538</td><td>[19,994–176,612]</td></tr><tr><td>RU</td><td>Military Deaths</td><td>523</td><td>76,687</td><td>[38,670–139,772]</td></tr><tr><td>UA</td><td>Military Deaths</td><td>67</td><td>17,223</td><td>[6,219–39,105]</td></tr><tr><td>RU</td><td>Military Injuries</td><td>44</td><td>148,608</td><td>[45,749–365,649]</td></tr><tr><td>UA</td><td>Military Injuries</td><td>8</td><td>33,081</td><td>[5,260–125,925]</td></tr><tr><td>RU</td><td>MLRS</td><td>261</td><td>488</td><td>[148–1,222]</td></tr><tr><td>UA</td><td>MLRS</td><td>27</td><td>538</td><td>[155–1,482]</td></tr><tr><td>RU</td><td>Tanks</td><td>501</td><td>3,380</td><td>[1,704–6,178]</td></tr><tr><td>UA</td><td>Tanks</td><td>33</td><td>2,051</td><td>[385–5,946]</td></tr><tr><td>RU</td><td>UAVs</td><td>292</td><td>337</td><td>[153–707]</td></tr><tr><td>UA</td><td>UAVs</td><td>40</td><td>1,643</td><td>[387–4,371]</td></tr></table>Note that for some loss categories, Feb. 23, 2023, may be many months beyond the latest observed report. Loss types with few data or that represent composite categories are omitted for concision.DiscussionOverall, we ﬁnd that Russian and Ukrainian equipment losses are often comparable by category, but that Russian personnel losses outpace Ukrainian personnel losses. This may reﬂect accounts of poorly equipped Russian soldiers and ineffective supply lines leading to relatively higher human costs, a narrative that has been popular in the media. As of the one-year mark, Russia appears to have lost personnel relative to Ukraine at a rate of 5.53 to 1 (1.6:1–14.5:1).More generally, we have proposed a method for measuring conﬂict-related losses with high temporal ﬁdelity from open- source data. Our approach deals with source-speciﬁc biases in a principled way, treating them as parameters to be estimated. It also incorporates both daily and cumulative reports about multiple distinct loss categories, given as either ranges or point estimates. This allows researchers to leverage the breadth of available reporting when reporting on any single type of loss is likely to be scarce.Materials and MethodsWe use a single multivariate Bayesian model with two outcomes: daily and cumulative loss counts. Every observation consists of a loss report (either daily or cumulative), the loss report’s “source,” the country to which the loss refers(the “target,” either Russia or Ukraine), the category of the loss (e.g., tanks, helicopters), the day of the reported loss, and whether the report is a range (e.g., “50–100”),lowerbound(e.g.,“atleast50”),upperbound, or a pointestimate.We assume that the outcomes are either Poisson- or negative binomial–distributed with means that are log-linear in covariates. For every category–target pair (e.g., “Tanks-Ukraine”), we also estimate a latent time series of expected daily losses using cubic basis splines.We assume Poisson and negative binomial distributions for daily andcumulative losses, yidaily and yjcum, respectively, where i and j index dailyand cumulative observations (Eqs. 1 &amp; 2). The log daily and cumulative meanestimates are shown in Eqs. 3 and 4. Coefﬁcients are denoted with 훽 . Ourestimates of the latent time series for every loss category for every target countryare represented by 휃ct,d, where ct indexes the category–target and d the numberofdayssinceFebruary24,2022.The mean dailylossesfora givencategory–target,훽<img src="/media/202408//1724856293.9117508.png" />nst, are drawn from a normal distribution (Eq. 6).This mean is added to a timetrend, 훽<img src="/media/202408//1724856293.952831.png" />end (d/365), and to time-varying deviations, B훽tpline, to capture thetemporal dynamics of losses, where 훽tpline are spline coefﬁcients and B is a cubicbasis spline with 150 kn. Spline coefﬁcients are multivariate normal distributed(Eq. 7). Every claim source (indexed by s) is given two scalar coefﬁcients toaccount for minimum and maximum estimates when ranges are given (Eq. 8).Iimin and Iimax indicate that observation i is a minimum or maximum estimate.Every observation is scaled by a multilevel bias coefﬁcient that is speciﬁcto itssource–target pair (indexed by st) and category, 훽<img src="/media/202408//1724856293.960546.png" />,<img src="/media/202408//1724856293.997323.png" />s, to account for systematicover- or underestimation. These are normally distributed around source–targetspeciﬁc means which are themselves normally distributed with prior mean zero(Eq. 9). We assume zero-centered bias terms, encoding the conservative beliefthat a source is unbiased absent data to indicate otherwise. A nonzero meanwould encode belief in a systematic bias (e.g., systematic measurement error).We model bias terms hierarchically to mitigate class imbalances through partial pooling. When estimating losses, we set the bias terms to zero (i.e., “no bias”). Category–target-speciﬁcoverdispersion is accounted for by 휙ct (Eq. 10). For brevity, we omit hyperpriors.yidaily ~ Pois(exp(휇idaily)), [<a href="#bookmark2">1</a>]yjcum NB(exp(휇jcum), 1/exp(휙ct[j])), [<a href="#bookmark3">2</a>]휇idaily = 휃ct[i],d[i] + 훽<img src="/media/202408//1724856294.06034.png" />[iia]<img src="/media/202408//1724856294.153292.png" />st[i] + 훽[ii]nIimin + 훽[ia]xIimax, [<a href="#bookmark1">3</a>]휇jcum = ln(Σ=[j]1exp(휃ct[j],d[k])) + 훽<img src="/media/202408//1724856294.225536.png" />[jia]<img src="/media/202408//1724856294.3025951.png" />st[j] + 훽[ji]nIjmin + 훽[ja]xIjmax,[4]휃ct,d = (B훽tpline)d + 훽<img src="/media/202408//1724856294.336782.png" />nst + 훽<img src="/media/202408//1724856294.366185.png" />end (d/365) , [5]<table><tr><td>Priors훽<img src="/media/202408//1724856294.395914.png" />end ~ N(휇trend, 휎trend)훽<img src="/media/202408//1724856294.4052181.png" /> ~ N(휇 min, 휎 min)훽<img src="/media/202408//1724856294.456867.png" />,<img src="/media/202408//1724856294.4613109.png" />s ~ N(훾sbtias, 휎<img src="/media/202408//1724856294.4661078.png" />tias)</td><td>훽<img src="/media/202408//1724856294.5891452.png" /> ~ N(휇const, 휎const)훽tpline ~ N(0, Σspline),훽 ~ N(휇 max, 휎 max),훾sbtias ~ N(0, 휎1bias),</td><td>[6] [7] [8] [9]</td></tr></table>휙ct ~ N(휇휙, 휎휙 ), [10]1. Russia calls up 300,000 reservists, says 6,000 soldiers killed in Ukraine. Reuters (2022).2. Ministry of Defense of Ukraine, The total combat losses of the enemy from 24.02 to 22.09. Ministry of Defense of Ukraine Ofﬁcial Website (2022).3. N. Sambanis, What is civil war? Conceptual and empirical complexities of an operational deﬁnition. J. Conﬂ. Res. 48, 814–858 (2004).4. B. Lacina, Explaining the severity of civil wars. J. Conﬂ. Res. 50, 276–289 (2006).5. H. Hegre, P. Vesco, M. Colaresi, Lessons from an escalation prediction competition. Int. Interactions 48, 521–554 (2022).6. R. Ruggles, H. Brodie, An empirical approach to economic intelligence in WorldWar II. J. Am. Stat. Assoc. 42, 72–91 (1947).7. L. J. Klarevas,C. Gelpi,J. Reiﬂer, Casualties,polls, and the Iraq War. Int. Sec. 31, 186–198 (2006).8. H. Cooper, E. Schmitt, Ukraine War Plans Leak Prompts Pentagon Investigation (The New York Times, 2023).9. M. Gibilisco,J. Steinberg, Strategic reporting: A formal model of biases in conﬂict data. Am. Polit. Sci. Rev., 1–17 (2022).10. C. von Clausewitz, M. Howard, P. Paret, On War (Princeton University Press, 2008).11. Z. Obermeyer,C. J. L. Murray, E. Gakidou, Fifty years of violent war deaths from Vietnam to Bosnia: Analysis of data from the world health survey programme. BMJ 336, 1482–1486 (2008).12. M. Spagat,A. Mack,T. Cooper,J. Kreutz, Estimating war deaths: An arena of contestation. J. Conﬂ. Res. 53, 934–950 (2009).13. B. Lacina, N. P. Gleditsch, Monitoring trends in global combat: A new dataset of battle deaths. Eur. J. Popul. 21, 145–166 (2005).14. A. Khorram-Manesh, K. Goniewicz, F. M. Burkle,Y. Robinson, Review of military casualties in modern conﬂicts—The re-emergence of casualties from armored warfare. Mil. Med. 187, e313–e321 (2021).Downloaded from https://www.pnas.org by 5 1. 195.24 1.227 on December 3, 2023 from IP address <a href="51.195.241.227">51.195.241.227</a>.PNAS 2023 Vol. 120 No. 34 e2307372120 <a href="https://doi.org/10.1073/pnas.2307372120">https://doi.org/10.1073/pnas.2307372120</a> 3 of 3