Monitorings of Russian disinformation topics
Methodology
Data collection and preprocessing
We collect news pieces from a sample of the following websites: (1) Ukrainian clickbait sites, (2) Russian sites aimed at Ukraine, (3) mainstream Russian news sites, and (4) mainstream Ukrainian news sites. We monitor only manipulative news from the first two group of sites, and all pieces from mainstream Russian and Ukrainian sites. Topics and news on Ukrainian mainstream sites are given for comparison, the majority of them are not manipulative.
Data is loaded from sites’ RSS-feeds. Each news item is loaded with its time of publishing, title, fulltext, and a link. We selected only items written in Russian since more disinformation is written in this language. Each items’ text was prepared for analysis — tokenized (split words and punctuation), lemmatized (words converted to normal form, infinitives).
We analyze only news about politics, economy, society, and external affairs of Ukraine. News about weather, celebrities, sports, car accidents, and similar are not included in monitoring and treated as irrelevant. The distinction between relevant and irrelevant news items is made by separate classification algorithm
Detection of manipulative news
Each news item was evaluated by an improved version of the manipulative news classifier: the algorithm was further trained on new data to improve accuracy (previous version of the classifier). It estimates the likelihood that the news contains emotional manipulation and/or false argumentation. According to news from clickbait sites, sites from the occupied territories and publications with the anti-Ukrainian position, the classifier finds 62% of the materials containing at least one type of manipulation, while incorrectly marks as manipulative 6% of the materials. That is, the algorithm rather misses the manipulation than falsely marks it.
In groups of Ukrainian manipulative sites there are only sites where algorithm has classified more than 10% of all relevant items as manipulative. From this Ukrainian clickbait sites and Russian sites targeting Ukraine we analyze only manipulative news pieces. Topics on mainstream sites are displayed for all news items, they are not checked by the classifier of manipulative news.
Sites in monitoring
- 72 russian sites, targeting Ukraine (only manipulative items): 3652.ru, 3654.ru, 8692.ru, anna-news.info, antifashist.com, antimaydan.info, c-inform.info, comitet.su, crisis.in.ua, delovoydonbass.ru, dnr-lnr.info, dnr-pravda.ru, dnr24.com, dnr24.su, donbasstoday.ru, doneck-news.com, dontimes.news, dontimes.ru, dosie.su, e-gorlovka.com.ua, e-news.su, evening-crimea.com, free-news.su, fresh.org.ua, fromdonetsk.net, front-novorossii.ru, gorlovka.today, jankoy.org.ua, kafanews.com, komtv.org, kv-journal.su, lgt.su, luga1news.ru, lugansk1.info, meridian.in.ua, metayogg.com, miaistok.su, mir-lug.info, mnyug.com, mozaika.dn.ua, nahnews.org, naspravdi.info, nefakt.info, newc.info, news-front.info, newsland.com, nk.org.ua, novorosinform.org, novorossiy.info, novosti.icu, odnarodyna.org, on-line.lg.ua, patriot-donetsk.ru, pohnews.org, politnavigator.net, politobzor.net, pravdanews.info, ruinformer.com, rusdnepr.ru, rusnext.ru, russian-vesna.ru, rusvesna.su, sevastopol.su, sevnews.info, sobytiya.info, svodki24.ru, time-news.net, ukraina.ru, voenkor.info, voskhodinfo.su, vsednr.ru, xvesti.ru
- 104 Ukrainian online publications where over 10% of all news about Ukraine have been spotted as being manipulative (manipulative materials only): 0432.in, 112.ua, agrimpasa.com, aif.ua, akcenty.com.ua, antikor.com.ua, argumentua.com, baza-pravda.in.ua, bbcccnn.com.ua, begemot.media, bessarabiainform.com, censoru.net, dialog.ua, donetskie.com, elise.com.ua, expres.life, figurant.com.ua, finoboz.net, for-ua.life, fraza.ua, from-ua.com, glavcom.life, glavk.info, glavred.info, glavred.life, golos.ua, hpib.life, hyser.com.ua, inforesist.org, inform-ua.info, informator.news, intell.in.ua, ivasi.news, jizn.info, job-sbu.org, k-stenke.com, khersonline.net, kompromat1.info, kompromat1.news, kordon.org.ua, korr.com.ua, kyiv.press, lifedon.com.ua, mignews.com.ua, monitor-ua.com, newnews.in.ua, news247.com.ua, news24ua.com, newser.com.ua, newsmir.info, newsua.one, nezalezhna.com, onpress.info, ord-02.com, ostrovok.lg.ua, p-zona.com, podrobnosti.ua, politdengi.com.ua, politeka.net, politica.com.ua, pravda.rv.ua, prioritet.org, proua.com.ua, provokators.top, prportal.com.ua, replyua.net, rezzonans.in.ua, rnbee.com.ua, rupor.info, rupor.od.ua, rusjev.net, sharij.net, skelet-info.org, spektrnews.in.ua, spichka.news, spzh.news, startko.co.ua, strana.ua, t.ks.ua, tema.in.ua, timer-odessa.net, tp0432.in.ua, tribunanaroda.info, ua-reporter.com, ua-vestnik.com, ua24ua.net, ukr.life, ukraine-zmi.info, ukrainianwall.com, ukranews.com, ukranews.life, ukrpost.biz, ukrrudprom.ua, vesti-ukr.com, vesti.ua, vkulake.com, vlasti.net, vremya.com.ua, vybor.ua, vz.ua, xn--j1aidcn.org, zhitomir-online.com, zik.ua, znaj.ua
- 17 Ukrainian online publications (all materials): 24tv.ua, bykvu.com, censor.net.ua, fakty.com.ua, fakty.ua, gordonua.com, interfax.com.ua, lb.ua, liga.net, nv.ua, pravda.com.ua, rbc.ua, segodnya.ua, tsn.ua, ukrinform.ru, unian.net, zn.ua
- 16 major Russian publications (all materials): aif.ru, dni.ru, kommersant.ru, kp.ru, lenta.ru, lentainform.com, life.ru, newsru.com, pravda.ru, regnum.ru, riafan.ru, russian.rt.com, slovodel.com, svpressa.ru, tass.ru, vz.ru
- 175 Ukrainian local publications (all materials): 0432.ua, 048.ua, 04849.com.ua, 04868.com.ua, 0512.com.ua, 056.ua, 0564.ua, 057.ua, 061.ua, 06153.com.ua, 0619.com.ua, 06239.com.ua, 06274.com.ua, 06277.com.ua, 0629.com.ua, 0642.ua, 1kr.ua, 1news.zp.ua, 44.ua, 5632.com.ua, 5692.com.ua, 62.ua, 6264.com.ua, 7kanal.com.ua, actual.today, allkharkov.ua, atn.kharkov.ua, barometr.info, beg.dp.ua, bessarabiainform.com, city.sumy.ua, cxid.info, dancor.sumy.ua, dani-info.com, dnepr.com, dneprcity.net, dneprnews.com.ua, dneprnews.info, dnews.dn.ua, dnl.com.ua, dnl.dn.ua, dnpr.com.ua, donbass.ua, donetskie.com, dontimes.news, dozor.kharkov.ua, dp.vgorode.ua, dpchas.com.ua, dumskaya.net, dv-gazeta.info, dzerghinsk.org, ecity.cn.ua, gloss.ua, golos.zp.ua, gorod-online.net, gorod.cn.ua, gorod.dp.ua, gx.net.ua, inform.zp.ua, informator.dp.ua, inshe.tv, insidernews.info, irtafax.com.ua, iz.com.ua, izbirkom.org.ua, izmailvechernii.com.ua, izmnews.info, izvestia.kharkov.ua, kg.ua, kh.vgorode.ua, kharkov.dozor.ua, khersonline.net, khustinfo.com/video, kiev.informator.ua, kiev.vgorode.ua, kievpravda.com, kn.ks.ua, kolokol.od.ua, korabelov.info, kramatorsk.info, kramatorskpost.com, krlife.com.ua, krnews.ua, krug.com.ua, kstati.dp.ua, lenta.kharkiv.ua, lugradar.net, mariupol-express.com.ua, mariupol.in, mariupolnews.com.ua, mig.com.ua, misto.zp.ua, mk.mk.ua, most-dnepr.info, mrpl.city, mycity.mk.ua, mykharkov.info, nashemisto.dp.ua, nashkiev.ua, nashreporter.com, nefakt.info, news.mspravka.info, newsroom.kh.ua, nf.dp.ua, niknews.mk.ua, nikolaevnews.com.ua, nikvesti.com, novosti-n.mk.ua, novosti-n.org, novosti.dn.ua, novosti.zhitomir.ua, objectiv.tv, od-news.com, od.vgorode.ua, odessa-life.od.ua, odessa.net.ua, odessa.online, odessapost.com, on.od.ua, ostro.org, otkat.od.ua, podrobnosti.mk.ua, press-center.news, prichernomorie.com.ua, pro.berdyansk.biz, rama.com.ua#tab_3-widget_panorama_posts_feed-2, real-vin.com, reni-odessa.od.ua, reporter-ua.com, ru.korzonews.info, rupor.od.ua, ruporzt.com.ua, russkiev.info, slavdelo.dn.ua, sobytiya.dp.ua, sq.com.ua, subbota.ua, svsever.lg.ua, t.ks.ua, telegraf.in.ua, timenews.in.ua, timer-odessa.net, timeua.info, tnua.info, topnews.cn.ua, topnews.odessa.ua, topnews.zp.ua, topnews.zt.ua, topor.od.ua, treebuna.info, tribun.com.ua, tribune.com.ua, tvplus.dn.ua, ua-reporter.com, uanews.kharkiv.ua, usionline.com, verge.zp.ua, vesti.dp.ua, visti.ks.ua, vmestezp.org, vn.mk.ua, vo.od.ua, volnorez.com.ua, vseprokiev.com.ua, vv.com.ua, x1.com.ua, z-city.com.ua, zachem.com.ua, zaodessu.com.ua, zhitomir-online.com, zhzh.info, zi.dn.ua, zp.vgorode.ua, zptown.zp.ua, zv.zp.ua
Topic detection
Selected manipulative news, 3,000 pieces per week on average, was broken down into topics of the week by automatic topic modeling (NMF). We edited the resulting news clusters manually: similar topics combined, irrelevant or overly general clusters discarded, topics that did not follow Russian disinformation deleted. Topics were identified automatically, so a small part of the news may be unrelated to the topic.
Next, we assigned each weekly topic a general topic and trained a classifier to predict the topic of news items (if any). Topics are detected automatically, so there is a small number of incorrectly classified items in each topic. For the dashboard home page general topics were united in meta-topics, which represent key narratives of disinformation.
Each subtopic is illustrated by a sample of headlines. For mainstream sites it is a random sample of all news on topic. A sample of headlines on Ukrainian clickbait sites and Russian sites targeting Ukraine is composed of materials classified as manipulative with high confidence.