Commit cd4a9c6b authored by Sacha's avatar Sacha Committed by SXibolet@2PITAU
Browse files

remove trash

parent 07e3a738
\documentclass[a4paper,12pt]{article}
\begin{document}
\pagestyle{empty}
\begin{enumerate}
\item Switch to conditional probability for linguistic features:
Recall $p(X_{sent} = x_i) = \frac{P(x_i) \in C'_i}{\sum_{i=1}^{z} f(x_i) \in C'_i}$ is equivalent to the number of comments (tweets) with sentiment $x_i$ divided by the total number of comments. Therefore, $p(X_{caps} = x_i) = p(X_{sent} = x_i | C'_i \in Caps) = \frac{p(X_{sent} = x_i \cap C'_i \in Caps)}{p(C'_i \in Caps)}$ where $Caps$ is the set of comments that have capitalized terms. This works similarly for the extreme lexicon.
With SentiStrength, sentiment scores range from $-4$ to $4$ thus $z=9$.
\end{enumerate}
\end{document}
pdflatex fa15-report.tex
open fa15-report.pdf
# -*- coding: utf-8 -*-
"""Creates representative API response for specification
"""
import json
if __name__ == "__main__":
with open('syria-2.json') as spec:
data = json.load(spec)
sample_data = data['result'][0]
sample_data['sentences'] = [sample_data['sentences'][0]]
sample_data['sentences'][0]['tweets'] = [sample_data['sentences'][0]['tweets'][0]]
with open('syria-spec.json', 'w') as new_spec:
str_rep = json.dumps(sample_data,
separators=(',', ': '),
indent=2)
new_spec.write(str_rep)
This diff is collapsed.
{
"training": false,
"byline": "By TIM ARANGO",
"lead": "A question has come to dominate the lives of thousands of Syrians here: Embark on a perilous journey to Europe or build a new life in Turkey.",
"title": "Syrian Migrants in Istanbul Confront Choice: Stay or Move On",
"url": "http://www.nytimes.com/2015/12/24/world/europe/syrian-migrants-in-istanbul-confront-choice-stay-or-move-on.html",
"abstract": "A question has come to dominate the lives of thousands of Syrians here: Embark on a perilous journey to Europe or build a new life in Turkey.",
"xlarge": "https://www.nytimes.com/images/2015/12/21/world/21ISTANBUL1/21ISTANBUL1-articleLarge.jpg",
"source": "The New York Times",
"score": 140.29687018903874,
"full": "ISTANBUL \u2014 At Pages, a bookstore and cafe in a three-story wooden house in a tidy cul-de-sac up a hill in this city\u2019s oldest quarter, a literary refuge has blossomed for young Arabs in exile \u2014 mainly Syrians but also Iraqis, Libyans and Yemenis who have fled the troubles of home and taken up residence here. The shelves are lined with Arabic translations of Orhan Pamuk, Jack Kerouac, Gabriel Garc\u00eda M\u00e1rquez and Franz Kafka. Two Syrian novels are particularly popular: \u201cNo Knives in This City\u2019s Kitchens,\u201d about a family living under tyranny, and \u201cThe Shell,\u201d about a prisoner in a Syrian jail. \u201cThis is the generation of revolution in all of the Arab world,\u201d said Samer Alkadri, a painter and graphic designer from Damascus, Syria, who opened the cafe with his wife in July. He stocks Arabic books he has flown in from a publishing house in Beirut, Lebanon. \u201cThey want to know why everything happened, and they don\u2019t believe the media.\u201d The cafe, and the little garden out back where a lazy rabbit visits on many days, is also a place to discuss the one question that has come to dominate the lives of Syrians here: to stay, or to embark on the perilous sea journey to Europe. That question has become more urgent in the wake of a recent deal between Turkey and the European Union aimed at stemming the flow of asylum seekers to the Continent. As Europe confronts a crisis in which nearly 700,000 people this year have made a journey that started with a dangerous sea passage from Turkey to Greece, it has turned to Turkey for help. The European Union, under the recent pact, is offering more than $3 billion in aid to the Turkish government to spend on improving the lives of Syrians, such as building schools and health facilities, with the hope that many would then choose to stay and build their lives here. Four of the five people who worked for Mr. Alkadri when he opened the cafe are now in Europe. Mr. Alkadri, though, has vowed to stay. \u201cI am here because I want to return to Syria,\u201d he said. \u201cMaybe tomorrow, or maybe after 10 years.\u201d \u201cAfter this terrible war,\u201d he added, \u201cwe will need people to rebuild Syria.\u201d For many Syrians, the decision will depend on whether Turkey can deliver on a promise to allow them to work legally, which could raise their wages and, as many Syrians hope, offer a way to citizenship. Basil Qali, 19, left a government-controlled suburb of Damascus in October, and on a recent morning he was sweeping the floor of a cafe here. He works 13 hours a day and earns the equivalent of about $280 a month. \u201cHere you work for about 13 hours a day only for money for food,\u201d he said. He said he would think about staying in Turkey if he could make more money and have a chance at citizenship. But for now, he thinks only of Europe. \u201cFor us, there is no point in talking about what is going on in Syria because we know it will last for five or 10 more years,\u201d he said. \u201cSo we mostly talk about a plan for getting to Europe, and making money and saving money.\u201d Another young Syrian in Istanbul, Mohammed Sayid, 20, works as a restaurant deliveryman. He said his father had been a supporter of the rebellion in Syria, and had been executed by the government. He has learned Turkish and hopes to stay here, provided the government does more to improve his life by allowing him to work legally. He complains of discrimination \u2014 girls he flirts with in bars turn away, he said, when they learn he is Syrian \u2014 but he is hesitant to join his friends in Europe. \u201cI\u2019ve been here as a stranger for a long time,\u201d Mr. Sayid said. \u201cI\u2019m not ready to be a stranger again in a new country.\u201d Despite a recent exodus, there are still nearly 2.2 million Syrian refugees in Turkey, the most of any host country in the region. In Istanbul alone, there are roughly 350,000, writing their own chapter in the history of a city that has long been a place of exile. Over the years, they have become woven into the frenetic life of this city, opening cafes, performing as street musicians on its busiest boulevard and panhandling on street corners, holding out Syrian passports to elicit the sympathies of passers-by. Perhaps not since the end of World War I, when Muslims were fleeing Greece and the Balkans, and Russians were escaping the Bolshevik Revolution, have Istanbul\u2019s streets swelled with so many \u00e9migr\u00e9s. Syrians here say they are newly optimistic because of the victory of President Recep Tayyip Erdogan\u2019s Islamist Justice and Development party in parliamentary elections last month. Syrians, by and large, love Mr. Erdogan because of Turkey\u2019s longstanding open-door policy for refugees, and ahead of the election they worried that opposition parties hostile to foreigners would do well at the ballot box. \u201cHe opened the doors for Syrians,\u201d Mr. Alkadri said. Mr. Erdogan and his allies consider themselves heirs to the Ottoman past, when Istanbul \u2014 then Constantinople \u2014 ruled an empire that stretched to the Arab world and into Europe. It was also the seat of the Muslim caliphate, but the lands they controlled were a mosaic of ethnicities and faiths. The modern Turkey that emerged from the collapse of the empire at the end of World War I was a largely homogeneous country based on an ethnic understanding of national identity. Political groups that see themselves as the upholders of this tradition \u2014 rivals to Mr. Erdogan and his party \u2014 have been hostile to the presence of so many Syrians. Now, without an election to muddy the political waters, and with billions in European money for the government to spend, many Syrians here are hopeful that Mr. Erdogan\u2019s government will see to their basic needs, which are many. In a recent report, for example, Human Rights Watch said that 400,000 Syrian children in Turkey were not in school. At Pages, Mohammed Kayali, 32, works on promoting the cafe on social media. He already made the journey to Europe, to Sweden in 2012, but came back. \u201cSurvivor\u2019s guilt,\u201d he said, simply, when asked why. And, he added, \u201cThe weather.\u201d When friends contemplating the journey ask his advice, Mr. Kayali tells them, \u201cIn some ways you are going to regret it.\u201d In Europe, he said that because of generous social programs, \u201cYou are guaranteed a living but not a life.\u201d By that, he meant a loss of identity. \u201cYou will be forced to change too much.\u201d On the night of the elections last month, when it became clear that Mr. Erdogan\u2019s party was cruising to victory, Mr. Kayali picked up his phone and called his fianc\u00e9e, also in Turkey. \u201cWe are going to stay,\u201d he told her.",
"sentences": [
{
"tweets": [
{
"is_negative": false,
"is_positive": false,
"sentiment": 0,
"author": "kamalkaroum",
"retweeted": false,
"ts": "Thu Jan 07 00:38:05 +0000 2016",
"n_statuses": 6353,
"pimg": "http://pbs.twimg.com/profile_images/681907688712843264/Mc0EEWbj_normal.jpg",
"time_zone": "Casablanca",
"followers": 2208,
"location": "",
"clean": "syria here",
"identifier": 684896754479792128,
"retweets": 0,
"dirty": "#syria here https://t.co/63rTJeN4EG"
}
],
"text": "\u201cThis is the generation of revolution in all of the Arab world,\u201d said Samer Alkadri, a painter and graphic designer from Damascus, Syria, who opened the cafe with his wife in July.",
"score": 6.186682698996645,
"linguistic_score": 5.155639062229566,
"sentiment_score": 1.031043636767078
}
],
"sentiment_score": 31.710930212630974,
"linguistic_score": 108.58593997640777,
"published": "2015-12-27"
}
\ No newline at end of file
{
"keyword": "clinton",
"ok": 1,
"result": [
{
"abstract": "At a time when liberals are ascendant in the party, some Democrats believe Hillary Rodham Clinton\u2019s time representing Wall Street as a senator could become a vulnerability.",
"byline": "By PATRICK HEALY",
"comments": [
{
"abuseFlag": null,
"clean": "Good article. I am glad to see the NYT covering this. Although this article did not mention the $25 million of \"speaking fees\" that Clinton accepted just in 2014. Most of the money she as to her name, she got from rich people who paid her to talk. Does anyone really believe that she earned millions of dollars for just talking for an hour? The \"speaking fees\" are legalized bribes!",
"dirty": "Good article. I am glad to see the NYT covering this. Although this article did not mention the $25 million of \"speaking fees\" that Clinton accepted just in 2014. Most of the money she as to her name, she got from rich people who paid her to talk. Does anyone really believe that she earned millions of dollars for just talking for an hour? The \"speaking fees\" are legalized bribes!",
"is_negative": false,
"is_positive": true,
"n_recommendations": 0,
"n_replies": 0,
"sentiment": 1,
"ts": "1448236950",
"userLocation": "MI"
}, ...
],
"full": "John Wittneben simmered as he listened to Hillary Rodham Clinton defend her ties to Wall Street during last weekend\u2019s Democratic debate. He lost 40 percent of his savings in individual retirement accounts during the Great Recession, while Mrs. Clinton has received millions of dollars from the kinds of executives he believes should be in jail. \u201cPeople knew what they were doing back then, because of greed, and it caused me harm,\u201d said Mr. Wittneben, the Democratic chairman in Emmet County, Iowa. \u201cWe were raised a certain way here. Fairness is a big deal.\u201d The next day he endorsed Senator Bernie Sanders in the presidential race. Mrs. Clinton\u2019s windfalls from Wall Street banks and other financial services firms \u2014 $3 million in paid speeches and $17 million in campaign contributions over the years \u2014 have become a major vulnerability in states with early nomination contests. Some party officials who remain undecided in the 2016 presidential race see her as overly cozy with big banks and other special interests. At a time when liberals are ascendant in the party, many Democrats believe her merely having \u201crepresented Wall Street as a senator from New York,\u201d as Mrs. Clinton reminded viewers in an October debate, is bad enough. It is an image problem that she cannot seem to shake. Though she criticizes the American economy as being \u201crigged\u201d for the rich, Mrs. Clinton has lost some support recently from party members who think she would go easy on Wall Street excess if elected. Even as she promises greater regulation of hedge funds and private equity firms, liberals deride her for refusing to support reinstatement of the Glass-Steagall Act, a law that separated commercial and investment banks until its repeal under President Bill Clinton. (Mr. Sanders favors its restoration.) And for many Democrats, her strong support from wealthy donors and a big-money \u201csuper PAC\u201d undercuts her increasingly progressive rhetoric on free trade and other economic issues. Her advisers say most Democrats like her economic policies and believe she would fight for middle-class and low-income Americans. Most opinion polls put Mrs. Clinton well ahead of Mr. Sanders nationally and in Iowa, and they are running even in New Hampshire, but she fares worse than him on questions about taking on Wall Street and special interests. And even if Mrs. Clinton sews up the nomination quickly, subdued enthusiasm among the party\u2019s liberal base could complicate efforts to energize Democratic turnout for the general election. In the primaries, Mrs. Clinton\u2019s advisers privately concede that she will lose some votes over her Wall Street connections. They declined to share specific findings from internal polls, but predicted the issue could resonate in Democratic contests in Iowa, Nevada, Ohio and Michigan, where many have lost homes and businesses to bank foreclosures. Mr. Sanders zeros in on Wall Street donations to Mrs. Clinton in an aggressive new television commercial that started running in Iowa and New Hampshire on Saturday: \u201cThe truth is, you can\u2019t change a corrupt system by taking its money,\u201d he warns. One of Mrs. Clinton\u2019s most prominent supporters in Ohio, former State Senator Nina Turner, defected to Mr. Sanders this month in part, she said, because she felt he would be tougher on special interests. And some Democratic superdelegates, whose backing is crucial, said Mrs. Clinton\u2019s ties to big banks, and her invocation of 9/11 to defend her ties to Wall Street at the Nov. 14 debate, only made them further question her independence from the financial industry. \u201cMy parents had a saying in Spanish \u2014 \u2018Dime con qui\u00e9n andas y te dir\u00e9 qui\u00e9n eres\u2019 \u2014 which means, \u2018Tell me who you\u2019re hanging with and I\u2019ll tell you who you are,\u2019\u201d said Alma R. Gonzalez, an uncommitted superdelegate from Florida. \u201cA lot of my Democratic friends feel that way about Hillary and Wall Street. \u201cAre the working people in this country going to be able to count on hard decisions being made by President Hillary Clinton with regard to her Wall Street chums?\u201d Ms. Gonzalez continued. \u201cWill she be another President Clinton who appoints a Treasury secretary from Wall Street? These are major concerns.\u201d Indeed, Mr. Clinton\u2019s close relationships with Wall Street executives like Robert E. Rubin of Goldman Sachs, whom he named his Treasury secretary, and his support for undoing parts of Glass-Steagall have contributed to misgivings about Mrs. Clinton. Mrs. Clinton has proposed imposing risk fees on unwieldy big banks and empowering regulators to break them up if necessary \u2014 though this is not the wholesale breakup that Mr. Sanders favors under a return of Glass-Steagall. She also proposes to make sure fines for corporate wrongdoing hit executive bonuses, and to pursue criminal prosecutions when justified. On Saturday, Mrs. Clinton discussed her plans for regulating the financial industry, at a Democratic rally in North Charleston, S.C., where she spoke after Mr. Sanders and another candidate, former Gov. Martin O\u2019Malley of Maryland. \u201cI have the toughest, most comprehensive proposals for dealing with Wall Street,\u201d she began, and a voter holding a \u201cBernie\u201d sign interrupted, \u201cBy taking their money!\u201d Mrs. Clinton continued, saying her plan would address not just Wall Street banks, but also insurance companies and the \u201cshadow banking\u201d industry of hedge funds and private equity firms. \u201cI am willing to break up the big banks, if they need to be broken up,\u201d she added. Yet even though she has taken tough stands in the past, such as chastising banks for widespread foreclosures in 2007 and 2008, some Democrats are skeptical that she would ever crack down hard on the executives in her social circles in Manhattan, the Hamptons and Washington. Jake Quinn, an uncommitted superdelegate from North Carolina, said he was concerned about Mrs. Clinton\u2019s willingness to clamp down on Wall Street malfeasance. \u201cThe financial sector\u2019s ongoing relative lack of accountability makes me suspicious of any candidate who sources it for significant support,\u201d he said. Mrs. Clinton\u2019s advisers say that any political harm resulting from her Wall Street ties will be minimal because she never took action in exchange for donations and that her proposals discredit claims that she would protect banks as president. The advisers also play down the possibility that Mrs. Clinton will face problems with voter enthusiasm and turnout if she is nominated. While Mr. Sanders and Mr. O\u2019Malley have argued that big donors inevitably had influence with her, her campaign has pushed back against suggestions that the financial services industry has bankrolled her campaign. Her aides also said ads by a new group, Future 45, attacking Mrs. Clinton would underscore her independence, because the group\u2019s major donors include Wall Street magnates like Paul Singer. \"When billionaire hedge fund managers are forming super PACs to run ads attacking her, it\u2019s clear they fear she will take action as president to crack down on the industry\u2019s abuses,\" said Brian Fallon, a Clinton campaign spokesman. Bashing Wall Street is not an automatic win for Mr. Sanders, however. Ms. Gonzalez, the Florida superdelegate, and some other undecided Democrats said they viewed Mr. Sanders as too hostile to banks and corporations and too divisive in his remarks about American wealth. But others said they were more concerned that Mrs. Clinton had not broken with Wall Street in a clear way, noting the lengths she went to at the debate to explain the relationship. \u201cShe was waving the bloody shirt of 9/11 to defend herself, which we\u2019re accustomed to seeing with demagogues on the right, and it just didn\u2019t feel quite right,\u201d said Kurt Meyer, a co-chairman of the Mitchell County Democrats in Iowa, who has not endorsed a candidate. \u201cShe connected two things, 9/11 and her ties to Wall Street, that I didn\u2019t like her sewing together.\u201d Ms. Turner, the former Ohio lawmaker, said the blocks of foreclosed homes in Cleveland were a painful reminder that banks prioritize their own corporate interests. Mr. Sanders has been criticizing \u201cthe corrupt economy symbolized by Wall Street greed\u201d for decades, she said. \u201cHe shows righteous indignation and speaks for the common woman and man in saying they have a right to be outraged at Wall Street,\u201d Ms. Turner said. \u201cHe doesn\u2019t just talk the talk. He walks the talk.\u201d And Mrs. Clinton? \u201cHer ties are her ties,\u201d Ms. Turner said.",
"lead": "At a time when liberals are ascendant in the party, some Democrats believe Hillary Rodham Clinton\u2019s time representing Wall Street as a senator could become a vulnerability.",
"linguistic_score": 109.79326659822776,
"published": "2015-11-22",
"score": 153.78719125576572,
"sentences": [
{
"linguistic_score": 3.2169171866886996,
"score": 4.390660392428369,
"sentiment_score": 1.1737432057396702,
"text": "He lost 40 percent of his savings in individual retirement accounts during the Great Recession, while Mrs. Clinton has received millions of dollars from the kinds of executives he believes should be in jail.",
"tweets": [
{
"author": "KWQCAshley",
"clean": ". says \"It's great to be back in Clinton\"",
"dirty": ".@HillaryClinton says \"It's great to be back in Clinton\" @kwqcnews https://t.co/k5EWWM9Iej",
"followers": 1129,
"identifier": 668595626041606144,
"is_negative": false,
"is_positive": true,
"location": "Davenport, IA",
"n_statuses": 6717,
"pimg": "http://pbs.twimg.com/profile_images/658138092822360069/B4c3knyO_normal.png",
"retweeted": false,
"retweets": 0,
"sentiment": 1,
"time_zone": "Central Time (US & Canada)",
"ts": "Mon Nov 23 01:03:13 +0000 2015"
}, ...
]
}, ...
]
}
]
}
{
"error": 0,
"result": {
"controversial": [
{
"EntropyScore": 2689.61,
"Term": "clinton"
},
{
"EntropyScore": 2005.47,
"Term": "bush"
},
{
"EntropyScore": 1580.31,
"Term": "putin"
},
{
"EntropyScore": 1359.08,
"Term": "romney"
},
{
"EntropyScore": 1174.0,
"Term": "hungary"
},
{
"EntropyScore": 1125.78,
"Term": "syria"
}
],
"top-5": [
"clinton",
"hippo",
"romney",
"austria",
"greece"
],
"trending": {
"AIDs": 3.571428571428571,
"afghanistan": 3.571428571428571,
"african": 3.571428571428571,
"austria": 3.571428571428571,
"baltimore": 3.571428571428571,
"bush": 3.571428571428571,
"clinton": 3.571428571428571,
"germany": 3.571428571428571,
"greece": 3.571428571428571,
"hillary": 3.571428571428571,
"hillary clinton": 3.571428571428571,
"hippo": 3.571428571428571,
"hulu": 3.571428571428571,
"hungary": 3.571428571428571,
"illinois": 3.571428571428571,
"iran": 3.571428571428571,
"iraq": 3.571428571428571,
"isis": 3.571428571428571,
"new york": 3.571428571428571,
"new york city": 3.571428571428571,
"obama": 3.571428571428571,
"prince": 3.571428571428571,
"princeton": 3.571428571428571,
"putin": 3.571428571428571,
"romney": 3.571428571428571,
"ron paul": 3.571428571428571,
"stanford": 3.571428571428571,
"syria": 3.571428571428571
}
}
}
\ No newline at end of file
{
"error": 0,
"result": [
{
"EntropyScore": 1359.08,
"Performed": "Tue, 13 Oct 2015 16:45:51 GMT"
},
{
"EntropyScore": 488.701,
"Performed": "Wed, 11 Nov 2015 17:46:48 GMT"
},
{
"EntropyScore": 572.734,
"Performed": "Sun, 22 Nov 2015 21:39:06 GMT"
}
]
}
\ No newline at end of file
{
"error": 0,
"result": {
"queries": [
{
"Performed": "Tue, 20 Oct 2015 18:30:12 GMT",
"Term": "clinton"
},
{
"Performed": "Tue, 20 Oct 2015 18:30:31 GMT",
"Term": "bush"
},
{
"Performed": "Tue, 27 Oct 2015 18:58:16 GMT",
"Term": "obama"
},
{
"Performed": "Tue, 27 Oct 2015 19:00:02 GMT",
"Term": "iraq"
},
{
"Performed": "Tue, 27 Oct 2015 19:03:57 GMT",
"Term": "afghanistan"
},
{
"Performed": "Wed, 28 Oct 2015 02:37:38 GMT",
"Term": "baltimore"
},
{
"Performed": "Wed, 28 Oct 2015 02:52:47 GMT",
"Term": "new york"
},
{
"Performed": "Wed, 28 Oct 2015 02:53:56 GMT",
"Term": "new york city"
},
{
"Performed": "Wed, 28 Oct 2015 03:22:28 GMT",
"Term": "iran"
},
{
"Performed": "Wed, 28 Oct 2015 03:24:13 GMT",
"Term": "stanford"
},
{
"Performed": "Wed, 28 Oct 2015 13:49:12 GMT",
"Term": "princeton"
},
{
"Performed": "Wed, 28 Oct 2015 16:19:07 GMT",
"Term": "greece"
},
{
"Performed": "Wed, 28 Oct 2015 17:17:06 GMT",
"Term": "germany"
},
{
"Performed": "Wed, 28 Oct 2015 17:26:48 GMT",
"Term": "hungary"
},
{
"Performed": "Wed, 28 Oct 2015 18:00:53 GMT",
"Term": "austria"
},
{
"Performed": "Wed, 28 Oct 2015 21:43:20 GMT",
"Term": "putin"
},
{
"Performed": "Tue, 10 Nov 2015 18:30:26 GMT",
"Term": "illinois"
},
{
"Performed": "Tue, 10 Nov 2015 18:31:39 GMT",
"Term": "prince"
},
{
"Performed": "Wed, 11 Nov 2015 17:46:35 GMT",
"Term": "romney"
},
{
"Performed": "Wed, 11 Nov 2015 17:48:23 GMT",
"Term": "african"
},
{
"Performed": "Wed, 11 Nov 2015 17:50:38 GMT",
"Term": "hulu"
},
{
"Performed": "Wed, 11 Nov 2015 18:12:59 GMT",
"Term": "ron paul"
},
{
"Performed": "Sat, 21 Nov 2015 18:42:05 GMT",
"Term": "AIDs"
},
{
"Performed": "Sat, 21 Nov 2015 18:50:27 GMT",
"Term": "isis"
},
{
"Performed": "Sat, 21 Nov 2015 18:56:29 GMT",
"Term": "syria"
},
{
"Performed": "Sat, 21 Nov 2015 18:57:57 GMT",
"Term": "hillary"
},
{
"Performed": "Sat, 21 Nov 2015 19:00:40 GMT",
"Term": "hillary clinton"
},
{
"Performed": "Mon, 23 Nov 2015 11:23:42 GMT",
"Term": "hippo"
}
]
}
}
\ No newline at end of file
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment