Impact Assessment

What impact do information products have on people beyond simple frequency based metrics?

We have been developing, implementing, evaluating and applying a theory-driven, computational solution for assessing the impact of issue-focused information products on people, groups and society. To this end, we capture, model and analyze the map of stakeholders and themes associated with the main issues addressed in a piece of information (baseline model) and identify changes in the baseline over time (Diesner, Kim, & Pak, 2014). We have shown that our approach captures the measurement of most of the impact goals as defined by funders and practitioners in the impact assessment field, and that it can also complement and enhance the findings obtained by using traditional methods, e.g., surveys and focus group studies, that are common in this application domain. In our initial work, we focused on measuring (changes in) public awareness and consumer attitudes (Diesner & Rezapour, 2015).

Extending this work, we built a theory-driven framework and probabilistic prediction model for identifying different types of micro level impact (i.e., impact on individuals), such as changes versus reinforcement in personal behaviour, cognition and emotions (Rezapour & Diesner, 2017). This work involved the development of a theoretical model of micro level impact, the labelling of user-generated text data based on a codebook that we developed and evaluated, and applying supervised machine learning to the ground truth data. Using linguistic, lexical, and psychological features, we obtained an accuracy rate of 81% (F1) with a Random Forest classifier. In a related project, we used supervised machine learning to predict whether a review of an issue-focused information product was written by an expert or a layman (Jiang & Diesner, 2016); achieving 90.70% accuracy (F1 score) and learning that experts reviews feature more complex grammar and a higher diversity in their vocabulary, while layman reviews are more subjective and contextualized in peoples’ everyday lives. In general, we design and perform our feature construction and model analysis processes such that they allows us to explain text-based reasons for the observed differences between the learnt categories. This work has been supported by the FORD Foundation and the National Center for Supercomputing Applications (NCSA) at UIUC. We have closely collaborated with practitioners, e.g., film makers and authors, and funders in the impact assessment field on these projects.

References:

  1. Diesner, J., Kim, J., & Pak, S. (2014). Computational Impact Assessment of Social Justice Documentaries. Metrics for Measuring Publishing Value: Alternative and Otherwise, 17(3).
  2. Diesner, J., & Rezapour, R. (2015). Social Computing for Impact Assessment of Social Change Projects Social Computing, Behavioral-Cultural Modeling, and Prediction (pp. 34-43). Switzerland: Springer.
  3. Jiang, M., & Diesner, J. (2016). Says Who…? Identification of Critic versus Layman Reviews of Documentary Films. Proceedings of 26th International Conference on Computational Linguistics (COLING), Osaka, Japan.
  4. Rezapour, R., & Diesner, J. (2017). Classification and Detection of Micro-Level Impact of Issue-Focused Films based on Reviews. Proceedings of 20th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2017), Portland, OR.

Developed Technology:

ConText: A solultion to support text and network analysis.

Publications

Jiang M, Diesner J (2016) Says Who…? Identification of Critic versus Layman Reviews of Documentary Films. Proceedings of 26th International Conference on Computational Linguistics (COLING), Osaka, Japan

Jiang M, Diesner J (2016) Rating Prediction and Feature Comparison of Issue-Focused Documentaries versus Feature Films. Short paper at ACM Hypertext and Social Media, Halifax, Canada.

Diesner J, Kim J, Pak S (2014) Computational Impact Assessment of Social Justice Documentaries. Journal of Electronic Publishing (JEP), special issue Metrics for Measuring Publishing Value: Alternative and Otherwise. DOI: 10.3998/3336451.0017.306

Diesner J, Pak S, Kim J, Soltani K, Aleyasen A (2014) Computational Assessment of the Impact of Social Justice Documentaries. iConference, Berlin, Gemany. DOI: 10.9776/14125 (Acceptance Rate = 32%)

Diesner J, Kim J, Higgings A (2014) Socio-Semantic Network Analysis for Impact Assessment. Abstract European Social Networks Conference (EUSN), Barcelona, Spain.

Diesner J, Aleyasen A, Kim J, Mishra S, Soltani S (2013) Using Socio-Semantic Network Analysis for Assessing the Impact of Documentaries. Extended abstract WIN (Workshop on Information in Networks), New York, NY.

Diesner J, Kim J, Pak S (2014) Socio-Semantic Network Analysis for Impact Assessment of Documentaries.

Funder

FORD Foundation
#0155-0370: Computational impact assessment of issue focused media and information products
#0145-0558: Computational impact assessment of social justice documentaries and media
#0125-6162: Computational impact assessment of social justice documentaries

National Center for Supercomputing Applications Faculty Fellowship
Predictive modeling for impact asessment

Center for Investigative Reporting
Impact Assessment

Gifts from:

  • Peace is Loud
  • Shoot First Inc.
  • Picture Motion
  • VM People
  • Robert Stone Productions