itmove parsers to factolib.py - counterfacto - small software tool to analyze twitter and highlight counterfactual statements Err parazyd.org 70 hgit clone git://parazyd.org/counterfacto.git URL:git://parazyd.org/counterfacto.git parazyd.org 70 1Log /git/counterfacto/log.gph parazyd.org 70 1Files /git/counterfacto/files.gph parazyd.org 70 1Refs /git/counterfacto/refs.gph parazyd.org 70 1README /git/counterfacto/file/README.md.gph parazyd.org 70 1LICENSE /git/counterfacto/file/LICENSE.gph parazyd.org 70 i--- Err parazyd.org 70 1commit 5d141366aa5bc60d8ee62bc7229daa5b8d6d617b /git/counterfacto/commit/5d141366aa5bc60d8ee62bc7229daa5b8d6d617b.gph parazyd.org 70 1parent a6e2fee6da44bd214d6db79113be431142bb60bf /git/counterfacto/commit/a6e2fee6da44bd214d6db79113be431142bb60bf.gph parazyd.org 70 hAuthor: parazyd URL:mailto:parazyd@dyne.org parazyd.org 70 iDate: Thu, 9 Mar 2017 00:14:49 +0100 Err parazyd.org 70 i Err parazyd.org 70 imove parsers to factolib.py Err parazyd.org 70 i Err parazyd.org 70 iDiffstat: Err parazyd.org 70 i M .gitignore | 3 ++- Err parazyd.org 70 i M README.md | 31 +++++++++++++++++-------------- Err parazyd.org 70 i M counterfacto-web | 52 ++++++++++++++++++++++++++----- Err parazyd.org 70 i A factolib.py | 199 +++++++++++++++++++++++++++++++ Err parazyd.org 70 i M templates/index.html | 6 +++--- Err parazyd.org 70 i Err parazyd.org 70 i5 files changed, 266 insertions(+), 25 deletions(-) Err parazyd.org 70 i--- Err parazyd.org 70 1diff --git a/.gitignore b/.gitignore /git/counterfacto/file/.gitignore.gph parazyd.org 70 it@@ -1,2 +1,3 @@ Err parazyd.org 70 i credentials Err parazyd.org 70 i-twokenize.pyc Err parazyd.org 70 i+*.pyc Err parazyd.org 70 i+*.txt Err parazyd.org 70 1diff --git a/README.md b/README.md /git/counterfacto/file/README.md.gph parazyd.org 70 it@@ -1,5 +1,5 @@ Err parazyd.org 70 i- Err parazyd.org 70 i-# Counterfacto Err parazyd.org 70 i+Counterfacto Err parazyd.org 70 i+============ Err parazyd.org 70 i Err parazyd.org 70 i Counterfactual (noun) Err parazyd.org 70 i Err parazyd.org 70 it@@ -11,7 +11,7 @@ Effects: it starts off with disappointment, then one will be able to Err parazyd.org 70 i uncover insights or knowledge that can be used to enhance future Err parazyd.org 70 i performance, leading to a better outcome in life. Err parazyd.org 70 i Err parazyd.org 70 i----------------------------------------------------------------------------------- Err parazyd.org 70 i+----------------------------------------------------------------------- Err parazyd.org 70 i Err parazyd.org 70 i Counterfacto is a small software tool that can analyse search results Err parazyd.org 70 i on twitter to highlight counterfactual statements on certain topics. Err parazyd.org 70 it@@ -24,7 +24,8 @@ We deem such a tool as a useful experiment, considering the importance Err parazyd.org 70 i of counterfactual analysis for political sentiment assessments and Err parazyd.org 70 i focus on news stories. Err parazyd.org 70 i Err parazyd.org 70 i-## Dependencies Err parazyd.org 70 i+Dependencies Err parazyd.org 70 i+------------ Err parazyd.org 70 i Err parazyd.org 70 i Python is required along the following packages: Err parazyd.org 70 i Err parazyd.org 70 it@@ -43,7 +44,8 @@ After installing the necessary python modules, run `make`, which will Err parazyd.org 70 i then download the needed data for nltk, and tell you how to use your Err parazyd.org 70 i twitter credentials in counterfacto Err parazyd.org 70 i Err parazyd.org 70 i-### Running the web edition of counterfacto Err parazyd.org 70 i+Running the web edition of counterfacto Err parazyd.org 70 i+--------------------------------------- Err parazyd.org 70 i Err parazyd.org 70 i To run counterfacto along with its web interface, you will need Err parazyd.org 70 i additional dependencies: Err parazyd.org 70 it@@ -52,7 +54,8 @@ additional dependencies: Err parazyd.org 70 i python-flask Err parazyd.org 70 i ``` Err parazyd.org 70 i Err parazyd.org 70 i-## Usage Err parazyd.org 70 i+Usage Err parazyd.org 70 i+----- Err parazyd.org 70 i Err parazyd.org 70 i ``` Err parazyd.org 70 i usage: ./counterfacto [-a account] [-f tweetfile] [-s searchterm] Err parazyd.org 70 it@@ -64,17 +67,17 @@ The web interface can be ran with (port defaults to 5000): Err parazyd.org 70 i usage: ./counterfacto-web [-p port] Err parazyd.org 70 i ``` Err parazyd.org 70 i Err parazyd.org 70 i-## References Err parazyd.org 70 i- Err parazyd.org 70 i-- [Learning Representations for Counterfactual Inference (2016)](http://jmlr.org/proceedings/papers/v48/johansson16.pdf) Err parazyd.org 70 i- Err parazyd.org 70 i-- [Bounding and Minimizing Counterfactual Error (2016)](https://arxiv.org/abs/1606.03976) Err parazyd.org 70 i+References Err parazyd.org 70 i+---------- Err parazyd.org 70 i Err parazyd.org 70 i-- [Counterfactuals in the Language of Social Media: A Natural Language Processing Project in Conjunction with the World Well Being Project (2015)](http://www.seas.upenn.edu/~cse400/CSE400_2015_2016/reports/report_15.pdf) Err parazyd.org 70 i+* [Learning Representations for Counterfactual Inference (2016)](http://jmlr.org/proceedings/papers/v48/johansson16.pdf) Err parazyd.org 70 i+* [Bounding and Minimizing Counterfactual Error (2016)](https://arxiv.org/abs/1606.03976) Err parazyd.org 70 i+* [Counterfactuals in the Language of Social Media: A Natural Language Processing Project in Conjunction with the World Well Being Project (2015)](http://www.seas.upenn.edu/~cse400/CSE400_2015_2016/reports/report_15.pdf) Err parazyd.org 70 i Err parazyd.org 70 i-## Licensing Err parazyd.org 70 i+Licensing Err parazyd.org 70 i+--------- Err parazyd.org 70 i Err parazyd.org 70 i-Counterfacto is Copyright (C) 2016 by the Dyne.org Foundation Err parazyd.org 70 i+Counterfacto is Copyright (C) 2016-2017 by the Dyne.org Foundation Err parazyd.org 70 i as part of the PIEnews project Err parazyd.org 70 i Err parazyd.org 70 i Software written by Ivan J. Err parazyd.org 70 1diff --git a/counterfacto-web b/counterfacto-web /git/counterfacto/file/counterfacto-web.gph parazyd.org 70 it@@ -1,10 +1,37 @@ Err parazyd.org 70 i #!/usr/bin/env python2 Err parazyd.org 70 i+# Copyright (c) 2017 Ivan J. Err parazyd.org 70 i Err parazyd.org 70 i+import json Err parazyd.org 70 i+import sys Err parazyd.org 70 i from flask import Flask, render_template, request, json Err parazyd.org 70 i-import os Err parazyd.org 70 i+from twitter import * Err parazyd.org 70 i+ Err parazyd.org 70 i+import factolib Err parazyd.org 70 i+ Err parazyd.org 70 i+ Err parazyd.org 70 i+global tweetsFile Err parazyd.org 70 i+global taggedFile Err parazyd.org 70 i+ Err parazyd.org 70 i+taggedFile = "tagged.txt" Err parazyd.org 70 i+ Err parazyd.org 70 i+try: Err parazyd.org 70 i+ with open('credentials') as fd: Err parazyd.org 70 i+ exec(fd.read()) Err parazyd.org 70 i+except: Err parazyd.org 70 i+ print("no credentials file found. please create it") Err parazyd.org 70 i+ sys.exit(1) Err parazyd.org 70 i Err parazyd.org 70 i app = Flask(__name__) Err parazyd.org 70 i Err parazyd.org 70 i+def writetweets(tweets, twfile): Err parazyd.org 70 i+ twfile = open(twfile, "w") Err parazyd.org 70 i+ for s in tweets: Err parazyd.org 70 i+ sintweet = s["text"] Err parazyd.org 70 i+ sintweet = sintweet.replace("\n", " ") Err parazyd.org 70 i+ sintweet = sintweet.encode("ascii", "ignore") Err parazyd.org 70 i+ twfile.write(sintweet + "\n") Err parazyd.org 70 i+ twfile.close() Err parazyd.org 70 i+ Err parazyd.org 70 i @app.route("/") Err parazyd.org 70 i def main(): Err parazyd.org 70 i return render_template('index.html') Err parazyd.org 70 it@@ -17,10 +44,23 @@ def search(): Err parazyd.org 70 i if not _name or not _method: Err parazyd.org 70 i return "Wrong data. Please try again." Err parazyd.org 70 i Err parazyd.org 70 i+ api = Twitter(auth=OAuth(oatoken,oasecret,conskey,conssecret)) Err parazyd.org 70 i+ Err parazyd.org 70 i if _method == "account": Err parazyd.org 70 i- os.system("./counterfacto -a " + _name) Err parazyd.org 70 i+ statuses = api.statuses.user_timeline(screen_name=_name, count=100) Err parazyd.org 70 i+ tweetsFile = "fetchedtweets-" + _name + ".txt" Err parazyd.org 70 i+ writetweets(statuses, tweetsFile) Err parazyd.org 70 i+ factolib.classify(tweetsFile, taggedFile) Err parazyd.org 70 i+ Err parazyd.org 70 i elif _method == "searchterm": Err parazyd.org 70 i- os.system("./counterfacto -s " + _name) Err parazyd.org 70 i+ statuses = api.search.tweets(q=_name, count=1) Err parazyd.org 70 i+ tweetsFile = "fetchedsearch.txt" Err parazyd.org 70 i+ writetweets(statuses, tweetsFile) Err parazyd.org 70 i+ factolib.classify(tweetsFile, taggedFile) Err parazyd.org 70 i+ Err parazyd.org 70 i+ cfs = "counterfactuals.txt" Err parazyd.org 70 i+ with open(cfs) as f: Err parazyd.org 70 i+ return f.read() Err parazyd.org 70 i Err parazyd.org 70 i if __name__ == "__main__": Err parazyd.org 70 i try: Err parazyd.org 70 it@@ -29,7 +69,5 @@ if __name__ == "__main__": Err parazyd.org 70 i except: Err parazyd.org 70 i _port = 5000 Err parazyd.org 70 i Err parazyd.org 70 i- app.run( Err parazyd.org 70 i- host="0.0.0.0", Err parazyd.org 70 i- port=int(_port) Err parazyd.org 70 i- ) Err parazyd.org 70 i+ app.run(host="127.0.0.1", port=int(_port)) Err parazyd.org 70 i+ #subprocess.call(["xdg-open", "http://127.0.0.1:" + _port]) Err parazyd.org 70 1diff --git a/factolib.py b/factolib.py /git/counterfacto/file/factolib.py.gph parazyd.org 70 it@@ -0,0 +1,199 @@ Err parazyd.org 70 i+#!/usr/bin/env python2 Err parazyd.org 70 i+# Copyright (c) 2017 Ivan J. \n') Err parazyd.org 70 i+ Err parazyd.org 70 i+ form_vec.append(form) Err parazyd.org 70 i+ cf_count[form][0] += 1 Err parazyd.org 70 i+ tweet = tweetfile.readline() Err parazyd.org 70 i+ Err parazyd.org 70 i+ count = 0 Err parazyd.org 70 i+ for i in xrange(1, form_num): Err parazyd.org 70 i+ count += cf_count[i][0] Err parazyd.org 70 i+ Err parazyd.org 70 i+ print("Finished tagging...") Err parazyd.org 70 i+ tweetfile.close() Err parazyd.org 70 i+ taggedfile.close() Err parazyd.org 70 i+ Err parazyd.org 70 i+ print("counterfactuals: " + str(count) + "/100") Err parazyd.org 70 i+ counterfactuals.write("counterfactuals: " + str(count) + "/100
\n") Err parazyd.org 70 i+ counterfactuals.close() Err parazyd.org 70 1diff --git a/templates/index.html b/templates/index.html /git/counterfacto/file/templates/index.html.gph parazyd.org 70 it@@ -11,9 +11,9 @@ Err parazyd.org 70 i Err parazyd.org 70 i Err parazyd.org 70 i Err parazyd.org 70 i-
Err parazyd.org 70 i+ Err parazyd.org 70 i Err parazyd.org 70 i
Err parazyd.org 70 i

Counterfacto

Err parazyd.org 70 it@@ -28,7 +28,7 @@ Err parazyd.org 70 i
Err parazyd.org 70 i Err parazyd.org 70 i Err parazyd.org 70 i Err parazyd.org 70 i Err parazyd.org 70 .