tag:blogger.com,1999:blog-12872680940046272802024-03-14T01:45:36.921-07:00Socio-technologist Building technological support for social phenomena and studying social support for technological phenomena. Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.comBlogger17125tag:blogger.com,1999:blog-1287268094004627280.post-32827968170625930592017-05-24T13:32:00.003-07:002017-05-24T13:32:39.604-07:00Join my reddit AMA about Wikipedia and ethical, transparent AI<br />Hey folks, I'm doing an experimental Reddit AMA ("ask me anything") in <a href="https://www.reddit.com/r/IAmA/">r/IAmA</a> on June 1st at 21:00 UTC. For those who don't know, I <a href="https://www.mediawiki.org/wiki/ORES">create artificial intelligences that support the volunteers who edit Wikipedia</a>. I've been studying the ways that crowds of volunteers build massive, high quality information resources like Wikipedia for over ten years. This AMA will allow me to channel that for new audiences in a different (for us) way. I'll be talking about the work I'm doing with the ethics and transparency of the design of AI, how we think about <a href="https://en.wikipedia.org/wiki/Artificial_intelligence">artificial intelligence</a> on Wikipedia, and ways we’re working to counteract vandalism. I'd love to have your feedback, comments, and questions—preferably when the AMA begins, but also through the <a href="https://www.mediawiki.org/wiki/Talk:ORES">ORES talkpage on MediaWiki</a>.<br /><br />If you'd like to know more about what I do, see <a href="https://en.wikipedia.org/wiki/User:Halfak_(WMF)">my WMF staff user page</a>, <a href="https://www.wired.com/2015/12/wikipedia-is-using-ai-to-expand-the-ranks-of-human-editors/">this Wired piece about my work</a> or my paper, "<a href="http://www-users.cs.umn.edu/~halfak/publications/The_Rise_and_Decline/halfaker13rise-preprint.pdf">The Rise and Decline of an Open Collaboration System: How Wikipedia’s reaction to popularity is causing its decline</a>".Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-68174927926113630952016-10-20T11:23:00.004-07:002016-10-20T11:23:58.148-07:00Best practices for AI in the social spaces: Integrated refutations<div class="tr_bq">
So, I was listening to an NPR show titled "<a href="http://one.npr.org/?sharedMediaId=495753267:495753269">Digging into Facebook's File on You</a>". At some point, there was some casual discussion of laws that some countries in the European Union have re. users' ability to review and correct mistakes in data that is stored about them. This made me realize that <a href="https://meta.wikimedia.org/wiki/ORES">ORES</a> needs a good mechanism for you to review how the system classifies you and your stuff. </div>
<br />
As soon as I realized this, I wrote up a ticket describing my rough thoughts on what a "refutation" system would look like. From <a href="https://phabricator.wikimedia.org/T148700">https://phabricator.wikimedia.org/T148700</a>:<br />
<blockquote>
<span style="font-size: x-small;">Build a user-friendly UI for reviewing how ORES has classified you or any query-able slice of activity.</span><span style="font-size: x-small;"><br /></span><span style="font-size: x-small;">Include UI & API to refute classifications endpoints (via OAuth).</span><span style="font-size: x-small;"><br /></span><span style="font-size: x-small;">Include UI to curate "refutations" (suppress & meta-review).</span><span style="font-size: x-small;"><br /></span><span style="font-size: x-small;">Notes:</span><ul>
<li><span style="font-size: x-small;">Refutations need freeform text. Freeform text *must* have a suppression system.</span></li>
<li><span style="font-size: x-small;">We should include refutations in ORES query results.</span></li>
<li><span style="font-size: x-small;">We can borrow from MediaWiki's global user rights (ORES is global) for curation privs.</span></li>
</ul>
</blockquote>
On one hand, I'm excited by this idea because I think it will be a very interesting exploration into what kinds of feedback people want to give to ORES' classifications. I think it will provide a good means for humans to disagree with the machine in ways that are <i>effective</i>. It's important that we don't hide false-positive/negative reports somewhere that no one will ever see. It would be better if such reports were available as part of a query result -- so that ORES decision can be flagged and directly challenged by a human. <br />
<br />
On the other hand, I'm a little bit embarrassed that this wasn't part of the plan all along. I guess in a way, I expected the wiki to fill this infrastructure -- of critiquing ORES. But critiques on the wiki are hidden from the API that tools use to consume score and relegating them to a less effectual class of data. I feel a little bit stupid that it never occurred to me that humans should be able to affect ORES' output. <br />
<br />
Now to find the resources to construct this Meta-ORES system...Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com2tag:blogger.com,1999:blog-1287268094004627280.post-94389848751636462016-06-29T08:12:00.002-07:002016-06-29T08:12:52.673-07:00The end of "lowering barriers" as a metaphor for transition difficulty<span style="background-color: white; color: #222222; font-family: "arial" , sans-serif; font-size: x-small;">I remember sitting on the carpet at CSCW 2014 talking to <a href="https://ischool.syr.edu/people/directories/view/gmugar/">Gabriel Mugar</a> about boundaries and how people becoming aware of them. I've been working on some thoughts re. "barriers" being the wrong word because one imagines a passive wall of some height/permeability that can be removed or opened -- hence "lowered". </span><br />
<span style="background-color: white; color: #222222; font-family: "arial" , sans-serif; font-size: x-small;"><br /></span>
<span style="background-color: white; color: #222222; font-family: "arial" , sans-serif; font-size: x-small;">In the case of active socialization as some newcomer is welcomed to some established community, the "wall" is more than "lowered"; an active aid of conveyance across some threshold is provided. Here, <a href="https://en.wikipedia.org/wiki/Bruno_Latour">Latour's door opener </a>doesn't just make it easy to enter a building but also communicates that you are welcome. If we were to apply the barrier metaphor of a "wall" that might be lowered, we might imagine a simple hole in the wall being the lowest barrier. But when thinking about active socialization, a hole in the wall with a "welcomer" is more open. Does the "welcomer" lower the "wall" further? No. Suddenly the idea of "lowering as barrier" gets in the way of thinking clearly about the difficulty people experience when transitioning from "outside" to "inside" of a community. </span>Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-11937166947408139252016-03-13T11:30:00.000-07:002016-03-13T11:30:30.680-07:00Brief discussion: The Effects and Antecedents of Conflict in FLOSS dev.Hey folks. For this weeks blogging, I'll be reading and thinking about:<br />
<blockquote class="tr_bq">
A Filippova, <a href="https://scholar.google.com/citations?user=yZStGxsAAAAJ&hl=en&oi=sra">H Cho</a>, <a href="http://dl.acm.org/citation.cfm?id=2820018">The Effects and Antecedents of Conflict in Free and Open Source Software Development</a>. <i>CSCW</i>, 2016</blockquote>
This isn't a review or really a summary -- just some thoughts I had that I want to share.<br />
<br />
<h3>
Transformational leadership</h3>
<blockquote class="tr_bq">
<span style="font-size: x-small;">See https://en.wikipedia.org/wiki/Transformational_leadership</span></blockquote>
<div>
Filippova et al. highlight a "transactional leadership" style as one of the factors mitigating the negative effects of high-conflict FOSS teams. </div>
<div>
<br /></div>
<div>
I don't really like the word "transformational" since, to me, it seems to fail to highlight the key meaning of the term -- collaboration. A transactional leader doesn't own the process of change, but rather works with others to put together a vision and enact a change with other committed individuals. It involves <i>convincing</i> other of a directional change or new effort rather than enforcing it through force or punishment/reward. I think the key insight that transactional leaders have is that they do not <i>own </i>the process of change. At most they can be a key contributor to it. There will need to be a conversation and <a href="https://tools.ietf.org/html/rfc7282">some sort of consensus</a> before real change can happen. In my practice as a leader of a small group of volunteers developing a FOSS project, I can't imagine operating any other way. I must lead by example -- by making a convincing case for what we should do and allowing myself to be convinced by other. The direction we set will be owned by all. I need my team to feel this ownership and shared identity. This is a trick when there are deadlines and a disagreement threatens delaying substantial work, but in my experience, deadlines aren't that dead and delays can be real opportunities to step back and ask why the disagreement exists in the first place. </div>
<div>
<br /></div>
<div>
In the end, if I can't convince you (gentle reader, my assumed collaborator) that my idea is <i>good</i> and <i>worth pursuing</i> then maybe it's not actually <i>good </i>or <i>worth pursuing. </i>In a way, I see transactional leadership style a lot like code review. If I want to reprogram our team structure/plans, then I should be able to get the changes reviewed and supported by others. In the process of review, we can increase our shared notion of norms/goals and make sure that the implemented changes are actually good!</div>
<div>
<br /></div>
<h3>
Different types of conflict</h3>
<div>
So, I've been studying conflict patterns in Wikipedia for a while, but I've never really dug into the literature about different types of conflict. Of course, it's obvious that there *are* different types. I've written about this in the past <a href="https://www.blogger.com/Halfaker,%20A.,%20Kittur,%20A.,%20Kraut,%20R.,%20&%20Riedl,%20J.%20(2009,%20October).%20A%20jury%20of%20your%20peers:%20quality,%20experience%20and%20ownership%20in%20Wikipedia.%20In%20Proceedings%20of%20the%205th%20International%20Symposium%20on%20Wikis%20and%20Open%20Collaboration%20(p.%2015).%20ACM.">around reverts in Wikipedia</a>, but it's much more useful to apply past through on the subject than to inject my own naive point of view. Luckily, Filippova et al. provide a nice summary of <i>Task conflict, Affective conflict, Process conflict, </i>and <i>Normative conflict.</i></div>
<div>
<i><br /></i></div>
<div>
<ul>
<li>Task conflict: Conflict about what needs to be done. E.g. do we engineer ORES to use celery workers or do we just plan to have a large pool of independent uwsgi threads? While this can be good in that it brings a diverse perspective of possible implementations, it also might turn into a religious battle over the Right Way To Do Things(TM).</li>
<li>Affective conflict: AKA drama. Conflict due to bad blood -- relationships between people might cause conflict regardless of any task disagreement. E.g. do we attribute the failure of a team member to complete a task to the complexity of the task or to their general incompetence? </li>
<li>Process conflict: Disagreements over how to do tasks. E.g. do we require code-review and non-self-merges for *everything* or are there some reasonable exceptions. On the Revision Scoring project, I generally make process declarations that go unchallenged and then we iterate whenever the process seems to be not working. So far, I wouldn't really say that this has escalated to "conflict" yet, but I could see how it might. </li>
<li>Normative conflict: Disagreements about "group function". E.g. do we generally pursue a caution-first strategy or an open-first strategy (everything is open until a problem arises vs. everything is closed until we know we can safely open it)? This is a discussion that I'd like to come back to in a future blog post as I'm very opinionated about how the norms of Wikipedia (bold inclusionism & openness) should be extended to the software development community around MediaWiki. </li>
</ul>
<div>
<br /></div>
</div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-14329237059309560392016-03-06T11:15:00.000-08:002016-03-06T11:17:12.910-08:00Revscoring 1.0 and some demos<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://upload.wikimedia.org/wikipedia/commons/thumb/7/7a/Revision_Scoring_as_a_Service_logo.svg/600px-Revision_Scoring_as_a_Service_logo.svg.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="https://upload.wikimedia.org/wikipedia/commons/thumb/7/7a/Revision_Scoring_as_a_Service_logo.svg/600px-Revision_Scoring_as_a_Service_logo.svg.png" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The "Revision Scoring" logo</td></tr>
</tbody></table>
Hey folks,<br />
<br />
I've mostly been traveling recently, but I have got some hacking in. There are a few things I want to share.<br />
<br />
<br />
<ol>
<li>I just released the <a href="http://pythonhosted.org/revscoring/">revscoring 1.0</a> -- revscoring is the library that powers <a href="https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service">ORES</a>. I developed it to make building and deploying machine learning models as services easier. </li>
<li>I wrote up an <a href="https://github.com/wiki-ai/editquality/blob/master/ipython/reverted_detection_demo.ipynb">ipython notebook</a> that demonstrates how to build a machine learning classifier to detect vandalism in Wikipedia.</li>
<li>I just finished writing up an <a href="https://github.com/wiki-ai/revscoring/blob/master/ipython/feature_engineering.ipynb">ipython notebook</a> that digs into how the feature extraction system in revscoring works. </li>
</ol>
<div>
<br /></div>
<div>
Next week, I hope to be showing you some more about <a href="http://socio-technologist.blogspot.com/2015/12/disparate-impact-of-damage-detection-on.html">the disparate impact of damage detectors on anons</a>. Stay tuned.</div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-56513786787698630412016-01-24T11:48:00.001-08:002016-01-24T11:48:59.856-08:00Notes on writing a Wikipedia Vandalism detection paperHey folks. I've been reviewing <a href="https://scholar.google.com/scholar?q=wikipedia+vandalism+detection&btnG=&hl=en&as_sdt=0%2C24">Wikipedia vandalism detection papers</a> -- which have been an active genre since ~2008. I'll be writing a more substantial summary of the field at some point, but for now, I just want to share some notes on what I (a Wikipedia vandalism detection practitioner) want to see in future work in this area.<br />
<h2>
Two thresholds: auto-revert & patrol</h2>
<div>
There are two things that I want to do with a vandalism prediction model. Auto-revert at an extremely high level of confidence (SMALL% false-positive -- e.g. 1% or 0.1%) and patrol everything that might be vandalism (LARGE% recall -- e.g. 95% or 97.5%). These <a href="https://www-users.cs.umn.edu/~halfak/publications/When_the_Levee_Breaks/geiger13levee-preprint.pdf">two modes</a> correspond to auto-revert bots (like <a href="https://en.wikipedia.org/wiki/User:ClueBot_NG">ClueBot NG</a>) and recent changes partolling performed by Wikipedia editors. These two thresholds represent basic values to optimize for that represent a <i>real </i>reduction in the amount of time and energy that Wikipedians need to spend patrolling for vandalism. </div>
<div>
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Precisionrecall.svg/563px-Precisionrecall.svg.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Precisionrecall.svg/563px-Precisionrecall.svg.png" width="218" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Truth space of a classifier model.<br />[[<a href="https://commons.wikimedia.org/wiki/File:Precisionrecall.svg">:commons:File:Precisionrecall.svg</a>]] </td></tr>
</tbody></table>
<h3>
Optimizing recall for anti-vandal bots</h3>
<div>
Anti-vandal bots are great in that they operate for free (short of development and maintenance), but they bust behave nicely around humans. A bot that reverts edits is potentially hazardous and so Wikipedians & ClueBot NG maintainers have settled on a 0.1% false-positive rate and <a href="https://en.wikipedia.org/wiki/User:ClueBot_NG#Statistics">claim</a> that they are able to detect 40% of all vandalism. They also claim that, at an older false-positive rate threshold of 1%, the bot was able to catch 55% of all vandalism. </div>
<div>
<br /></div>
<div>
So, vandalism prediction model scholars. Please tell me what recall you get at 1% and 0.1% false-positive rates. As this proportion goes up, humans will need to spend less time and energy reverting vandalism.</div>
<div>
<br /></div>
<h3>
Optimizing review-proportion for patrollers</h3>
<div>
We may never reach the day where anti-vandal bots are able to attain 100% recall. In the meantime, we need to use human judgement to catch everything else. But we can optimize how we make use of this resource (human time and attention) by minimizing how many edits humans will need to review in order to catch some large percentage of the vandalism -- e.g. 95% or 97.5%. </div>
<div>
<br /></div>
<div>
So, vandalism prediction model scholars. Please tell me what proportion of all edits your model must flag as vandalism in order to get 95 and 97.5% recall. As this proportion goes down, humans will need to spend less time and energy reviewing.</div>
<div>
<br /></div>
<h2>
Realtime is the killer use-case</h2>
<div>
This is more of a rant that a request for measurements. A lot of papers explore how much more fitness that they can get using post-hoc measures of activity around an edit. It's no surprise that you can tell whether or not an edit was vandalism easier once you can include "was it reverted?" and "did the reverting editor call it vandalism?" in your model. There's lots of discussion around how these post-hoc models could be used to clean up a print version of Wikipedia, but I'm pretty sure we're never going to do that (<a href="http://printwikipedia.com/">at least not really</a>). If we ever did try to reduce views of vandalized articles, we'd probably want do that in realtime. ;) </div>
<div>
<br /></div>
<div>
<br /></div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com3tag:blogger.com,1999:blog-1287268094004627280.post-24177838612358667632015-12-06T10:28:00.003-08:002015-12-06T10:28:51.478-08:00Disparate impact of damage-detection on anonymous Wikipedia editorsToday, I'm writing briefly about a problem that I expect to be studying and trying to fix over the course of the next few weeks. <br />
<br />
<b>The problem: </b>The damage detection models that <a href="http://socio-technologist.blogspot.com/2015/10/ores-hacking-social-structures-by.html">ORES</a> supports seems to be overly skeptical of edits by anonymous editors and newcomers. <br />
<br />
I've been looking at this problem for a while, but I was recently inspired by by the framing of <a href="https://en.wikipedia.org/wiki/Disparate_impact">disparate impact</a>. Thanks to <a href="http://jacob.thebault-spieker.com/">Jacob Thebault-Spieker</a> for suggesting I look at the problem this way. <br />
<blockquote class="tr_bq">
<i><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;">In </span><a href="https://en.wikipedia.org/wiki/United_States" style="background: none rgb(255, 255, 255); color: #0b0080; font-family: sans-serif; font-size: 14px; line-height: 22.4px; text-decoration: none;" title="United States">United States</a><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> </span><a href="https://en.wikipedia.org/wiki/Anti-discrimination_law" style="background: none rgb(255, 255, 255); color: #0b0080; font-family: sans-serif; font-size: 14px; line-height: 22.4px; text-decoration: none;" title="Anti-discrimination law">anti-discrimination law</a><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;">, the theory of </span><b style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;">disparate impact</b><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> holds that practices in employment, housing, or other areas may be considered discriminatory and illegal if they have a disproportionate </span><b style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;">"adverse impact"</b><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> on persons in a protected class.</span> </i> via Wikipedia's <a href="https://en.wikipedia.org/wiki/Disparate_impact">Disparate Impact</a> (CC-BY-SA 4.0)</blockquote>
So, let's talk about some terms and how I'd like to apply them to Wikipedia. <br />
<br />
<b>Disproportionate adverse impact. </b>The damage detection models that ORES supports are intended to focus attention on potentially damaging edits. Still human judgement is not perfect and there's lot of fun research that suggests that "recommendations" like this <a href="https://www.researchgate.net/profile/John_Tsibouklis/publication/200110346_Is_seeing_believing_how_recommender_system_interfaces_affect_users%27_opinions/links/0c96053b4f7a29c7fb000000.pdf">can affect people's judgement</a>. So by encouraging Wikipedia's patrollers to look a particular edit, we are likely also making them more likely to find flaws in that edit than if it was not highlighted by ORES. Having an edit rejected can <a href="https://meta.wikimedia.org/wiki/Research:The_Rise_and_Decline">demotivate the editor</a>, but it may be even more concerning that the rejection of content from certain types of editors may lead to coverage biases as the editors most likely to contribute to a particular topic may be discouraged or prevented from editing Wikipedia<br />
<br />
<b>Protected class. </b> In US law, it seems that this term is generally reserved for race, gender, and ability. In the case of Wikipedia, we don't know these demographics. They <i>could </i>be involved and I think they likely are, but I think that anonymous editors and newcomers should also be considered a protected class in Wikipedia. Generally, anonymous editors and newcomers are excluded from discussions and therefor subject to the will of experienced editors. I think that this has been having a substantial, negative impact on the quality and coverage of Wikipedia. To state it simply, I think that there are a collection of systemic problems around anonymous editors and newcomers that prevent them from contributing to the dominant store of human knowledge.<br />
<br />
So, I think I have a moral obligation to consider the effect that these algorithms have in contributing to these issues and rectifying them. The first and easiest thing I can do is remove the features <span style="font-family: "courier new" , "courier" , monospace;">user.age</span> and <span style="font-family: "courier new" , "courier" , monospace;">user.is_anon</span> from the prediction models. So I did some testing. Here's fitness measures (see <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve">AUC</a>) all of the edit quality models with the current and without-user features included.<br />
<br />
<table class="remarkup-table" style="background: rgb(191, 207, 218); border-spacing: 1px; color: black; font-family: 'Segoe UI', 'Segoe UI Web Regular', 'Segoe UI Symbol', Lato, 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; line-height: 19.63px; margin: 12px 0px 0px; word-break: normal;"><tbody style="margin-bottom: 0px;">
<tr><th style="background: rgb(248, 249, 252); border: 0px; margin: 0px; padding: 4px 6px; vertical-align: top;">wiki</th><th style="background: rgb(248, 249, 252); border: 0px; margin: 0px; padding: 4px 6px; vertical-align: top;">model</th><th style="background: rgb(248, 249, 252); border: 0px; margin: 0px; padding: 4px 6px; vertical-align: top;">current AUC</th><th style="background: rgb(248, 249, 252); border: 0px; margin: 0px; padding: 4px 6px; vertical-align: top;">no-user AUC</th><th style="background: rgb(248, 249, 252); border: 0px; margin: 0px; padding: 4px 6px; vertical-align: top;">diff</th></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">dewiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.900</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.792</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.108</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">enwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.835</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.795</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.040</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">enwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">damaging</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.901</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.818</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.083</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">enwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">goodfaith</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.896</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.841</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.055</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">eswiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.880</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.849</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.031</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">fawiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.913</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.835</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.078</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">fawiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">damaging</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.951</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.920</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.031</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">fawiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">goodfaith</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.961</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.897</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.064</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">frwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.929</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.846</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.083</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">hewiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.874</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.800</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.074</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">idwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.935</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.903</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.032</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">itwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.905</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.850</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.055</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">nlwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.933</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.831</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.102</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">ptwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.894</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.812</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.082</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">ptwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">damaging</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.913</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.848</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.065</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">ptwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">goodfaith</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.923</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.863</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.060</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">trwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.885</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.809</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.076</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">trwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">damaging</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.892</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.798</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.094</td></tr>
<tr><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">trwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">goodfaith</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.899</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.795</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.104</td></tr>
<tr style="margin-bottom: 0px;"><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">viwiki</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">reverted</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.905</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">0.841</td><td style="background: rgb(255, 255, 255); border: 0px; margin: 0px; padding: 3px 6px; vertical-align: top;">-0.064</td></tr>
</tbody></table>
<br />
<div>
So to summarize what this table tells us: We'll lose between 0.05 and 0.10 AUC per model which brings us from <i>beating the state of the art</i> to <i>not.</i> That makes the quantitative glands in my brain squirt some anti-dopamine out. It makes me want to run the other way. It's <i>really cool </i>to be able to say "we're beating the state of the art". But on the other hand, it's kind of lame to know "we're doing it at the expense of users who are most sensitive and necessary." So, I've convinced myself. We should deploy these models that look less fit by the numbers, but also reduce the disparate impact on anons and new editors. After all, the actual practical application of the model may very well actually be better despite what the numbers say.</div>
<div>
<br /></div>
<div>
But before I do anything, I need to convince my users. They should have a say in this. At the very least, they should know what is happening. So, next week, I'll start a conversation laying out this argument and advocating for the switch. </div>
<div>
<br /></div>
<div>
One final note. This problem may be a blessing in disguise. By reducing the fitness of our models, we have a new incentive to re-double our efforts toward finding alternative sources of signal to increase the fitness of our models. </div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com4tag:blogger.com,1999:blog-1287268094004627280.post-13801013683135724652015-11-22T10:25:00.004-08:002015-11-22T10:26:21.383-08:00Vision of ORES & ORES visionSo I've been working on a blog post for <a href="http://blog.wikimedia.org/">blog.wikimedia.org</a> about ORES. I talked about ORES a few weeks ago in <a href="http://socio-technologist.blogspot.com/2015/10/ores-hacking-social-structures-by.html">ORES: Hacking social structures by building infrastructure</a>, so check that out for reference. Because the WMF blog is relatively high profile, the Comms team at the WMF doesn't want to just lift my personal bloggings about it -- which makes sense. I usually spend 1-2 hours on this, so you get typos and unfinished thoughts.<br />
<br />
In this post, I want to talk to you about something that I think is really important when communication about what ORES is to a lay audience. <br />
<br />
<h2>
Visualizing ORES</h2>
<div>
The WMF Comms team is pushing me to make the topic of machine triage much more approachable to a broad audience. So, I have been experimenting with visual metaphors that would make kinds of things that ORES enables easier to understand. I like to make simple diagrams like the one below for the presentations that I give. </div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/ORES_edit_quality_flow.svg/800px-ORES_edit_quality_flow.svg.png" imageanchor="1" style="margin-left: auto; margin-right: auto; text-align: center;"><img border="0" height="231" src="https://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/ORES_edit_quality_flow.svg/800px-ORES_edit_quality_flow.svg.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><div style="text-align: left;">
<span style="font-size: 12.8px;">The flow of edits from The Internet to Wikipedia are highlighted by ORES </span></div>
<div style="text-align: left;">
<span style="font-size: 12.8px;">quality prediction models as "good", "needs review" and "damaging". </span></div>
</td></tr>
</tbody></table>
<h2>
ORES vision</h2>
<div>
But it occurs to me that a metaphor might be more appropriate. With the right metaphor, I can communicate a lot of important things through implications. With that in mind, I really like using Xray specs as a metaphor for what ORES does. It hits a lot of important points about what using ORES means -- both what makes it powerful and useful and also why we should be cautious when using it. <br />
<div>
</div>
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://atomictoasters.com/wp-content/uploads/2012/03/X-Ray-Specs.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://atomictoasters.com/wp-content/uploads/2012/03/X-Ray-Specs.jpg" height="320" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: left;">A clipping from an old magazine showing fancy sci-fi specs. </td></tr>
</tbody></table>
<div>
ORES shows you things that you couldn't see easily beforehand. Like a pair of Xray specs, ORES lets you peer into the firehose of edits coming into Wikipedia and see potentially damaging edits stand out in sharp contrast against the background of probably good edits. But just like a pair of sci-fi specs, ORES alters your perception. It implicitly makes subjective statements about what is important (separating the good from the bad) and it might bias you towards looking at the <b>potentially bad</b><i style="font-weight: bold;"> </i>with more scrutiny. While this may be the point, it can also be problematic. Profiling an editors work by a small set of statistics is inherently imperfect and the imperfections in the prediction can inevitably lead to biases. So I think it is important to realize that, when using ORES, you're perception is altered in ways that aren't simply <i>more truthful</i>.</div>
<div>
<br /></div>
<div>
So, I hope that the use of this metaphor will help educate ORES users in the level of caution they employ as this socio-technical conversation about how we <i style="font-weight: bold;">should</i><b> </b>use subjective, profiling algorithms as part of the construction of Wikipedia.</div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-35517135834647762872015-11-01T09:34:00.003-08:002015-11-01T09:34:56.068-08:00Measuring value-adding in WikipediaSo I've been working on this project on an off. I've been trying to bring robust measures of edit quality/productivity to Wikipedians. In this blog post, I'm going to summarize where I am with the project. <div>
<br /></div>
<div>
First, the umbrella project: <a href="https://meta.wikimedia.org/wiki/Research:Measuring_value-added">Measuring value-added</a></div>
<div>
<br /></div>
<div>
Basically, I see the value of Wikipedia as a simple combination of two hidden variables: <i>quality</i> and <i>importance</i>. If we focused on making our unimportant content really high quality, that wouldn't be very valuable. Conversely if we were to focus on increasing the <i>quality</i> of the most important content first, that would increase the <i>value</i> of Wikipedia most quickly. </div>
<blockquote class="tr_bq">
Value = Quality <b style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;">×</b> Importance</blockquote>
<div>
But I want to look at value-adding activities, so I need to measure progress towards quality. I think a nice term for that is <i>productivity</i>. </div>
<blockquote class="tr_bq">
Value-added = Productivity <b style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;">×</b> Importance</blockquote>
<div>
So in order to take measurements of value-adding activity in Wikipedia, I need to bring together good measure of <i>productivity </i>and <i>importance</i>.</div>
<h2>
Measuring importance</h2>
<div>
See <a href="https://meta.wikimedia.org/wiki/Research:Measuring_article_importance">https://meta.wikimedia.org/wiki/Research:Measuring_article_importance</a></div>
<div>
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d0/View_rate_density.by_wikiproject_importance.bounded.svg/640px-View_rate_density.by_wikiproject_importance.bounded.svg.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="228" src="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d0/View_rate_density.by_wikiproject_importance.bounded.svg/640px-View_rate_density.by_wikiproject_importance.bounded.svg.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><div style="text-align: left;">
<span style="font-size: 12.8px;">Density of log(view rate) for articles assessed by Wikipedians</span></div>
<div style="text-align: left;">
<span style="font-size: 12.8px;">for importance. </span></div>
</td></tr>
</tbody></table>
<div>
I'm going to side-step a big debate purely because I don't feel like re-hashing it in text. It's not clear what importance <i>is</i>. But we have some good ways to measure it. The two dominant strategies for determining the importance of a Wikipedia article's topic are (1) view rate counts and (2) link structure. </div>
<div>
<br /></div>
<div>
With view rate counts, the assumption is made that the most important content in Wikipedia is viewed most often. This works pretty well as far as assumptions go, but it has some notable weaknesses. For example, the article on <a href="https://en.wikipedia.org/wiki/Breaking_Bad">Breaking Bad</a> (TV show) has about an order of magnitude more views than the article on <a href="https://en.wikipedia.org/wiki/Chemistry">Chemistry</a>. For an encyclopedia of knowledge, it doesn't <i>feel right </i>that we'd consider a TV show to be more important than a core academic discipline. </div>
<div>
<br /></div>
<div>
Link structure provides another opportunity. Google's founders famously used the link structure of the internet to build a ranking strategy for the most important websites. See <a href="https://en.wikipedia.org/wiki/PageRank">PageRank</a>. This also seems to work pretty well, but it's less clear what the relationship is between the link graph properties and the nebulous notion of <i>importance</i>. At least with page view rates, you can plainly imagine the <i>impact</i> that a highly viewed article has. </div>
<div>
<br /></div>
<div>
Fun story though: Chemistry has 10 times as many incoming links as Breaking Bad. It could be that this measurement strategy can help us deal with the <i>icky </i>feeling us academics get when thinking that a TV show is more important than centuries of difficult work building knowledge. </div>
<h2>
Measuring productivity </h2>
<div>
See: <a href="https://meta.wikimedia.org/wiki/Research:Measuring_edit_productivity">https://meta.wikimedia.org/wiki/Research:Measuring_edit_productivity</a> <br /><div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://upload.wikimedia.org/wikipedia/commons/e/e7/Word_persistence.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="257" src="https://upload.wikimedia.org/wikipedia/commons/e/e7/Word_persistence.png" width="320" /></a></div>
<div>
Luckily, there is a vast literature for measuring the quality of contributions in Wikipedia. Many of which I have published! There are a lot of strategies, but the most robust (and difficult to compute) is tracking the persistence of content between revisions. The assumption goes: the more subsequent edits a contribution survives, the high quality it probably was. We can quite easily weight "words added" by "persistence quality" to get a nice productivity measure. It's not perfect, but it works. The trick is figuring out the right way to scale and weight the measures so that they are intuitively meaningful. </div>
<div>
The real trick here was making the computation tractable. It turns out that tracking changes between revisions is extremely computationally intensive. It would take me 60 days or so to track content persistence across the entire ~600m revisions of Wikipedia on a single core of the fastest processor on the market. So the trick is to figure out how to distribute the processing across multiple processors. We've been using Hadoop streaming. See my past post about it: <a href="http://socio-technologist.blogspot.com/2014/11/fitting-hadoop-streaming-into-my-python.html">Fitting Hadoop streaming into my workflow</a> It's been surprisingly difficult to work with memory issues in Hadoop streaming that don't happen when just using unix pipes on the command line. I might make a post about that later, but honestly, it just makes me feel tried to think about those types of problems. </div>
</div>
<h2>
Bringing it together</h2>
<div>
I'm almost there. I've still got to work out some threshholding bits for productivity measures, but I've already finished the hard computational work. My next update (or paper) will be about who, where and when of value-adding in Wikipedia. Until then, stay tuned. </div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-82714282853011793322015-10-26T07:01:00.004-07:002015-10-28T08:54:23.614-07:00My story of SQL as a data analysis equalizerSo, the title of this post is a little bit extreme, but I chose it specifically because I think it will help you (gentle reader) in thinking in the direction that I'd like you to think. <br />
<br />
Back when I was a young & bright eyed computer scientist who was slowly realizing that social dynamics were a fascinating and complex environment within which to practice technology development, I was invited to intern at the Wikimedia Foundation with a group of 7 other researchers. It turns out that we all had very different backgrounds going in. Fabian, <a href="http://www.whym.org/">Yusuke</a>, and I had a background in computer science. <a href="https://wikimediafoundation.org/wiki/User:Jtmorgan">Jonathan</a> had expertise in technology design methods (but not really programming). <a href="https://www.english.umd.edu/users/mkill">Melanie</a>'s expertise was in rhetoric and language. <a href="http://stuartgeiger.com/wordpress/">Stuart</a> was trained in sociology and philosophy of science (but he'd done a bit of casual programming to build some bots for Wikipedia). I think this diverse set of backgrounds enabled us to think very well about the problem that we had to face, but that's a subject for another blog entry. Today I want to talk to you about the technology that we ended up rallying around and taking massive advantage of: the Structured Query Language (SQL) and a Relational Database Management System (RDBMS).<br />
<br />
Up until my time at the Wikimedia Foundation, I had to do my research of Wikipedia the hard way. 1st, I downloaded Wikipedia's 10 terrabyte XML dump (compressed to ~100 gigabytes). Then I write Python script that used a streaming p7zip decompressor and a unix pipe to read the XML with a streaming processor. This workflow was complex. It tapped many of the skills I had learned in my training as a computer scientist. And yet, it was still incredibly slow to perform basic analyses. <br />
<br />
It didn't take me long to start using this XML processing strategy to produce intermediate datasets that could be loaded into a postgres RDBMS for high-speed querying. This was invaluable to making quick progress. Still I learned some lessons about including *all the metadata I reasonably could* in this dataset since going back to the XML was another headache and week-long processing job. As a side-note, I've since learned that many other computer scientists working with this dataset went through a similar process and have since polished and published their code that implements these workflows. TL;DR: This was a really difficult analysis workflow even for people with a solid background in technology. I don't think it's unreasonable to say that social scientist or rhetoric scholar would have found it intractable to do alone. <br />
<br />
When I was helping organize the work we'd be doing at the Wikimedia Foundation, I'd heard that there was a proposal in the works to get us researchers a replica of Wikipedia's databases to query directly. Honestly, I was amazed that people weren't doing this already. I put my full support behind it and thanks to others who saw the value, it was made reality. Suddenly I didn't need to worry about processing new XML dumps to update my analyses, I could just run a query against a database that was already indexes and up to date at all times. This was a breakthrough for me and I found myself doing explorations of the dataset that I had never considered before because the speed of querying the relevancy of the data made them possible. Stuart and I had a great time writing SQL queries for both our own curiosity and to explore what we ought to be exploring.<br />
<br />
For my coworkers who had no substantial background in programming, they saw yet another language that the techies were playing around with. So, they took advantage of us to help them answer their questions by asking us to produce excel-sized that they could explore. But as these things go when people gets busy, these requests would often remain unanswered for days at a time. I've got to hand it to Jonathan. Rather than twiddling his thumbs while he waited for a query request to be resolved, he decided to pick up SQL as a tool. I think he set an example. It's one thing to have a techy say to you, "Try SQL! It's easy and powerful." and a totally different thing for someone without such a background to agree. By the end of the summer internship, I don't think we had anyone (our managers included) who were not writing a bit of SQL here and there. All would agree that they were substantially empowered by it. <br />
<br />
Since then, Jonathan has made it part of his agenda to bring SQL and basic data analysis techniques to larger audiences. He's been a primary advocate (and volunteer product manager) of Quarry our new open querying to for Wikimedia data. That service has taken off like wildfire -- threatening to take down our MariaDB servers. Check it out https://quarry.wmflabs.org/ Specifically, I'd like to point you to the list of recent queries: https://quarry.wmflabs.org/query/runs/all Here, you can learn SQL techniques by watching others use them!Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-62183645940667829622015-10-04T14:46:00.002-07:002015-10-28T08:53:55.655-07:00ORES: Hacking social structures by building infrastructure So, I just crossed a major milestone on a system I'm building with <a href="https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service#Team">my shoe-string team</a> of mostly volunteers and I wanted to tell you about it. We call it ORES. <br />
<div style="text-align: right;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://upload.wikimedia.org/wikipedia/commons/thumb/5/51/Objective_Revision_Evaluation_Service_logo.svg/480px-Objective_Revision_Evaluation_Service_logo.svg.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="https://upload.wikimedia.org/wikipedia/commons/thumb/5/51/Objective_Revision_Evaluation_Service_logo.svg/480px-Objective_Revision_Evaluation_Service_logo.svg.png" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The ORES logo</td></tr>
</tbody></table>
<br />
The Objective Revision Evaluation Service is one part response to a feminist critique of power structures and one part really cool machine learning and distributed systems project. It's a machine learning service that is designed to take a very complex design space (advanced quality control tools for Wikipedia) and allow for a more diverse set of <a href="https://en.wikipedia.org/wiki/Standpoint_feminism">standpoints</a> to be expressed. I hypothesize that systems like these will make Wikipedia more fair and welcoming while also making it more efficient and productive. <br />
<h2>
Wikipedia's power structures</h2>
<div>
So... I'm not going to be able to go into depth here but there's some bits I think I can say plainly. If you want a bit more, see <a href="https://www.youtube.com/watch?v=Hj7o5d-OEis">my recent talk about it</a>. <b>TL;DR:</b> The technological infrastructure of Wikipedia was build through the lens of a limited standpoint and it was not adapted to reflect a more complete account of the world once additional standpoints entered the popular discussion. Basically, Wikipedia's quality control tools were designed for what Wikipedia editors needed in 2007 and they haven't changed in a meaningful way since. </div>
<h2>
Hacking the tools</h2>
<div>
I had some ideas on what kind of changes to the available tools would be important. In 2013, I started work in earnest on <a href="https://en.wikipedia.org/wiki/Wikipedia:Snuggle">Snuggle</a>, a <a href="https://civic.mit.edu/blog/natematias/supporting-change-from-outside-systems-with-design-and-data-stuart-geiger-on">successor system</a>. Snuggle implements a socialization support system that helps experienced Wikipedia editors find promising newcomers who need some mentorship. Regretfully, the project wasn't terribly successful. The system works great and I have a few users, but not as many the system would need to do its job at scale. In reflecting on this, I can see many reasons why, but I think the most critical one was that I couldn't sufficiently innovate a design that fit into the social dynamics of Wikipedia It was too big of a job. It requires the application of many different perspectives and a <i>conversation </i>of iterations. I was a PhD student -- one of the good ones because Snuggle gets regular maintenance -- but this work required a community.</div>
<div>
<br /></div>
<div>
When I was considering where I went wrong and what I should do next, I was inspired by was the sudden reach that Snuggle gained when the <a href="https://en.wikipedia.org/wiki/User:HostBot">HostBot</a> developer wanted to use my "promising newcomer" prediction model to invite fewer vandals to a new Q&A space. My system just went from 2-3 users interacting with ~10 newcomers per week to 1 bot interacting with ~2000 newcomers per week. Maybe I got the infrastructure bit right. Wikipedia editors <i>do </i>need the means to find promising newcomers to support after all!</div>
<h2>
Hacking the infrastructure</h2>
<div>
So, lately I've been thinking about infrastructure rather than direct applications of experimental technology. Snuggle and HostBot helped to know to ask the question, "What would happen if Wikipedia editors could find good new editors that needed help?" without imagining any one application. The question requires a much more <a href="http://socio-technologist.blogspot.com/2014/10/wikipedias-socio-technology-as-compared.html">system-theoretic way of reasoning about Wikipedia, technology and social structures</a>. Snuggle seemed to be interesting as an infrastructural support for Wikipedia. What other infrastructural support would be important and what changes might that enable across the system itself?</div>
<div>
<br /></div>
<div>
OK. Back to quality control tools -- the ones that haven't changed in the past 7 years despite the well known problems. Why didn't they change? Wikipedia's always had a large crowd of volunteer tool developers who are looking for ways to make Wikipedia work better. I haven't measured it directly, but I'd expect that this tech community is as big and functional as it ever was. There were loads of non-technological responses to the harsh environment for newcomers (including the <a href="https://www.blogger.com/"><span id="goog_1988251658"></span>Teahouse<span id="goog_1988251659"></span></a> and various WMF initiatives). AFAICT, the tool I built in 2013 was the *only* substantial technological response.</div>
<div>
<br /></div>
<div>
Why is there not a <i>conversation of innovation</i> happening around quality control tools? If you want to build a quality control tool for Wikipedia that works efficiently, you need a machine learning model that calls your attention to edits that are likely to be vandalism. Such an algorithm can reduce the workload of reviewing new edits in Wikipedia by 93%, but standing one up is excessively difficult. To do it well, you'll need an advanced understand of computer science and some substantial engineering experience in order to get the thing to work in real time. </div>
<div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://upload.wikimedia.org/wikipedia/commons/thumb/5/5c/Engineering_open_production_efficiency_at_scale_(Invited_talk_%40_Mizzou).pdf/page136-1024px-Engineering_open_production_efficiency_at_scale_(Invited_talk_%40_Mizzou).pdf.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="225" src="https://upload.wikimedia.org/wikipedia/commons/thumb/5/5c/Engineering_open_production_efficiency_at_scale_(Invited_talk_%40_Mizzou).pdf/page136-1024px-Engineering_open_production_efficiency_at_scale_(Invited_talk_%40_Mizzou).pdf.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The "activation energy" threshold to building a new quality<br />
control tool is primarily due to the difficulty of building a<br />
machine learning model.</td></tr>
</tbody></table>
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<br /></div>
<div>
So, <i>What would happen if Wikipedia editors could quickly find the good, the bad, and the newcomers in need of support. </i>I'm a computer scientist. I can build up infrastructure for that and cut the peak off of that mountain -- or maybe cut it down entirely. That's what ORES is. </div>
<div>
<br /></div>
<h2>
What ORES is</h2>
<div>
ORES is a web service that's provides access to a scale-able computing cluster full of state-of-the-art machine learning algorithms for detecting damage, differentiating good-faith edits from bad and measuring article quality. All that is necessary to use this service is to request a url containing the revision you want scored and the models you would like to apply to it. For example, if you wanted to know if <a href="https://en.wikipedia.org/wiki/?diff=190057686">my first registered edit</a> on Wikipedia was damaging, you could request the following URL. </div>
<div>
<br /></div>
<div>
<a href="https://ores.wmflabs.org/scores/enwiki/damaging/190057686/">https://ores.wmflabs.org/scores/enwiki/damaging/190057686/</a></div>
<div>
<br /></div>
<div>
Luckily, ORES does not think this is damaging in the slightest.</div>
<div>
<br /></div>
<div>
<blockquote class="tr_bq" style="white-space: pre-wrap; word-wrap: break-word;">
<span style="font-family: Courier New, Courier, monospace;">{<br /> "190057686": {<br /> "prediction": false,<br /> "probability": {<br /> "false": 0.9999998999999902,<br /> "true": 1.0000000994736041e-07<br /> }<br /> }<br />}</span></blockquote>
We use a distributed architecture to make scaling up the system to meet demand easy. The system is built in python. It uses <a href="http://celery.readthedocs.org/en/latest/">celery</a> to distribute processing load and redis for a cache. It is based on <a href="https://github.com/wiki-ai/revscoring">revscoring</a> library I wrote to generalize machine learning models of edits in Wikipedia. This same library will allow you to <a href="https://github.com/wiki-ai/ores-wikimedia-config/tree/master/models">download one of our model files</a> and use it on your own machine -- or just use our API. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://upload.wikimedia.org/wikipedia/commons/thumb/c/c0/ORES_request_flow_(cache%2Bcelery).svg/527px-ORES_request_flow_(cache%2Bcelery).svg.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="291" src="https://upload.wikimedia.org/wikipedia/commons/thumb/c/c0/ORES_request_flow_(cache%2Bcelery).svg/527px-ORES_request_flow_(cache%2Bcelery).svg.png" width="320" /></a></div>
<br />
<br />
Our latest models are substantially more fit than the state-of-the-art (<a href="https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CB4QFjAAahUKEwimmsab4qnIAhVHpR4KHUU7Dx0&url=http%3A%2F%2Frepository.upenn.edu%2Fcgi%2Fviewcontent.cgi%3Farticle%3D1515%26context%3Dcis_papers&usg=AFQjCNHEsZHGPbyNVrRvgJux-Eb2OcgVjg">.84 AUC</a> to our <a href="https://meta.wikimedia.org/wiki/ORES/damaging">.90-95 AUC</a>) our system has been substantially battle tested. Last month, <a href="https://en.wikipedia.org/wiki/Wikipedia:Huggle">Huggle</a>, one of the dominant yet unchanged quality control tools started making use of our service. We've seen tool devs and other projects leap at the ability to use our service.<br />
<ul>
<li><a href="https://tools.wmflabs.org/raun/?language=pt&project=wikipedia&userlang=en">Ra<span style="background-color: white; color: #222222; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 15px; line-height: 19.5px;">•</span>un</a> (for projects that support ORES)</li>
<li><a href="https://github.com/Krinkle/mw-gadget-rtrc/pull/43">Real-Time Recent Changes</a> </li>
<li><a href="https://en.wikipedia.org/wiki/Wikipedia:WikiProject_X">en:Wikipedia:WikiProject X</a> via <a href="https://en.wikipedia.org/wiki/User:Reports_bot">en:User:Reports bot</a></li>
<li><a href="https://en.wikipedia.org/wiki/Wiki_Education_Foundation">en:Wiki Education Foundation</a> – Student quality/productivity measurements</li>
<li><a href="https://en.wikipedia.org/wiki/User:SuggestBot">en:User:SuggestBot</a></li>
<li><a href="https://tools.wmflabs.org/crosswatch">crosswatch</a> – cross-wiki watchlist</li>
<li>(in development) <a href="https://gerrit.wikimedia.org/r/#/q/project:mediawiki/extensions/ORES,n,z">ORES extension to MediaWiki</a></li>
<li><a href="https://dashboard.wikiedu.org/recent-activity">WikiEd recent student activity dashboard</a></li>
<li><a href="https://outreachdashboard.wmflabs.org/article_finder">WikiEd article finder</a></li>
</ul>
</div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-21642102544427086232015-09-13T11:12:00.000-07:002015-09-13T11:21:06.843-07:00MediaWiki Utilities -- Unix styleToday, I want to talk about a specific type of research output that, I feel, adds a substantial amount of value beyond a single research project. I'm talking about open sourced research code, but not of the type that you normally see -- the type that's actually intended to be used by other people to serve new purposes. <br />
<br />
Before I dig into software libraries I want to talk to you about, I must make a distinction:<br />
<ul>
<li><a href="http://matt.might.net/articles/crapl/">CRAPL</a> quality code -- This is the code that a researcher builds ad-hoc in order to get something done. There's little thought spent on <i>generalizability</i> or <i>portability</i>. With this code, it's usually better and faster to fix a problem by adding a step in the workflow rather than fixing the original problem. So, you end up with quite a mess usually. While good for documenting the process of data science work, this is not <i style="text-decoration: underline;">useful</i> to others. </li>
<li>Library quality code -- This code has been designed to generalize to new problems. It's intended to be used as a utility by others who are doing similar work -- but not the exact same work. It's usually well documented and well tested. With this code, it is <i>sacreligious</i> to add more code on top of broken code to fix a problem. This is just one of the many disciplines that must be applied to the practice of writing software to have good, library quality code. </li>
</ul>
<br />
I've been analyzing Wikipedia data for nearly a decade (!!!) -- and I can tell you that it was never easy. The English Wikipedia XML dumps that I have done most of my work with are on the order of 10 terrabytes uncompressed. The database, web API and XML dumps all use different field names to refer to the same thing. In each one, the absence of a field -- or NULLing of the field can mean different things. Worse, the MediaWiki software has been changing over time, so in order to do historical analyses, you need to take that into account. In the process of working out these details and getting my work done, I've produced reams of CRAPL quality code. See https://github.com/halfak/Activity-sessions-research for an example. In this case, I have a Makefile that, if executed, would replicate my research project. But if you look inside that Makefile, you'll see things like this:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"># datasets/originals/enwiki_edit_action.tsv: sql/edit_action.sql<br /> # cat sql/edit_action.sql | \<br /> # mysql $(dbstore) enwiki > \<br /> # datasets/originals/enwiki_edit_action.tsv</span></blockquote>
<br />
That's a commented out Makefile rule that calls my local database with my local configuration hardcoded and runs some SQL against it. This is great if you want to know what SQL produced which datafile, but not very useful if you want to replicate the work. And why is it commented out!? Well, the database query takes a long time to run and I didn't want to accidentally overwrite the data file as I was finishing off the research paper. Gross, right? This isn't all that useful if you wanted to perform a similar analysis.<br />
<div>
<br />
But in producing this CRAPL code, there are some nice, generalizable parts that occur to me so I write them up for others' benefit. I've gone through a few iterations of this and learned from my mistakes. <br />
<br />
Back in 2011, I released the first version of <a href="https://bitbucket.org/halfak/wikimedia-utilities">wikimedia-utilities</a>, a set of utilities that made the work I was doing at the Wikimedia Foundation easier. The killer feature of this library was the XML processing strategy. It changed the work of processing Wikipedia's terrabyte scale XML dumps from a ~2000 line script to a ~100 line script. But the code wasn't very <a href="http://docs.python-guide.org/en/latest/writing/style/">pythonic</a>, it lacked proper tests and did not integrate well into the <a href="https://pypi.python.org/pypi">python packaging environment</a>. <br />
<br />
In 2013, I decided to make a clean break and start working on <a href="https://github.com/mediawiki-utilities/python-mediawiki-utilities">mediawiki-utilities</a>, a super-set of utilities from wikimedia-utilities that were intentionally generalized to run on any MediaWiki instance. I had learned some lessons about being pythonic, implementing proper tests and integrating with python's packaging environment.<br />
<br />
But as I had been working on new projects and realizing how they could generalize, I ended up expanding mediawiki-utilities to a monolith of loosely related parts. And it gets worse. Since I focused on those parts as I needed them, there were certain modules that were ignored. Since I did most of my work with the databases directly, it was rare that I spent time on the 'database' module of mediawiki-utilities. I ended up with a monolith that was inconsistently developed!<br />
<br />
So, in thinking about monoliths and how to solve problems that they impose, I was inspired by the <a href="https://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a> of combining "small, sharp tools" to solve larger problems. I realized that the primary modules of mediawiki-utilities could be split off into their own projects and combined in interesting ways -- and that this would enable a more distributed strategy to management. So I've been hard at work to bring this vision into the light.<br />
<br />
First, the core utilities:<br />
<ul>
<li><code style="background-color: #f9f9f9; border-radius: 2px; border: 1px solid rgb(221, 221, 221); font-family: monospace, Courier; font-size: 14px; line-height: 22.4px; padding: 1px 4px;">pip install mwxml</code><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://pythonhosted.org/mwxml" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">docs</a><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://github.com/mediawiki-utilities/python-mwxml" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">source</a></li>
<ul>
<li>This library contains a collection of utilities for efficiently processing MediaWiki’s XML database dumps. There are two important concerns that this module intends to address: complexity and performance of streaming XML parsing.</li>
</ul>
<li><code style="background-color: #f9f9f9; border-radius: 2px; border: 1px solid rgb(221, 221, 221); font-family: monospace, Courier; font-size: 14px; line-height: 22.4px; padding: 1px 4px;">pip install mwapi</code><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://pythonhosted.org/mwapi" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">docs</a><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://github.com/mediawiki-utilities/python-mwapi" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">source</a></li>
<ul>
<li>This library provides a set of basic utilities for interacting with MediaWiki’s “action” API – usually available at /w/api.php. The most salient feature of this library is the <a href="https://pythonhosted.org/mwapi/session.html#mwapi.Session">mwapi.Session</a> class that provides a connection session that sustains a logged-in user status and provides convenience functions for calling the MediaWiki API. See <a href="https://pythonhosted.org/mwapi/session.html#mwapi.Session.get">get()</a> and <a href="https://pythonhosted.org/mwapi/session.html#mwapi.Session.post">post()</a>.</li>
</ul>
<li><code style="background-color: #f9f9f9; border-radius: 2px; border: 1px solid rgb(221, 221, 221); font-family: monospace, Courier; font-size: 14px; line-height: 22.4px; padding: 1px 4px;">pip install mwdb</code><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://github.com/mediawiki-utilities/python-mwdb" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">source</a></li>
<ul>
<li>This library provides a set of utilities for connecting to and querying a MediaWiki database.</li>
</ul>
<li><code style="background-color: #f9f9f9; border-radius: 2px; border: 1px solid rgb(221, 221, 221); font-family: monospace, Courier; font-size: 14px; line-height: 22.4px; padding: 1px 4px;">pip install mwparserfromhell</code><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://mwparserfromhell.readthedocs.org/en/latest/" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">docs</a><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://github.com/earwig/mwparserfromhell" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">source</a></li>
<ul>
<li>This library provides an easy-to-use and outrageously powerful parser for MediaWiki wikicode.</li>
<li>Note that I am primarily a <i>user </i>of this library -- not a major contributor -- but it obviously belongs in this list.</li>
</ul>
</ul>
<div>
<br /></div>
</div>
<div>
Now the peripheral libraries that make use of these core utilities:</div>
<div>
<ul>
<li><code style="background-color: #f9f9f9; border-radius: 2px; border: 1px solid rgb(221, 221, 221); font-family: monospace, Courier; font-size: 14px; line-height: 22.4px; padding: 1px 4px;">pip install mwoauth</code><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://pythonhosted.org/mwoauth" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">docs</a><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://github.com/mediawiki-utilities/python-mwoauth" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">source</a></li>
<ul>
<li>This library provide a simple means to performing an OAuth handshake with a MediaWiki installation with the OAuth Extension installed.</li>
</ul>
<li><code style="background-color: #f9f9f9; border-radius: 2px; border: 1px solid rgb(221, 221, 221); font-family: monospace, Courier; font-size: 14px; line-height: 22.4px; padding: 1px 4px;">pip install mwreverts</code><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://pythonhosted.org/mwreverts" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">docs</a><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://github.com/mediawiki-utilities/mwreverts" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">source</a></li>
<ul>
<li>This library provides a set of utilities for detecting reverts (see <a href="https://pythonhosted.org/mwreverts/detection.html#mwreverts.Detector">mwreverts.Detector</a> and <a href="https://pythonhosted.org/mwreverts/detection.html#mwreverts.detect">mwreverts.detect()</a>) and identifying the reverted status of edits to a MediaWiki wiki.</li>
</ul>
<li><code style="background-color: #f9f9f9; border-radius: 2px; border: 1px solid rgb(221, 221, 221); font-family: monospace, Courier; font-size: 14px; line-height: 22.4px; padding: 1px 4px;">pip install mwsessions</code><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://pythonhosted.org/mwsessions/" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">docs</a><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> • </span><a class="external text" href="https://github.com/mediawiki-utilities/python-mwsessions" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">source</a></li>
<ul>
<li>This library provides a set of utilities for group MediaWiki user actions into sessions. <a href="https://pythonhosted.org/mwsessions/sessionization.html#mwsessions.Sessionizer">mwsessions.Sessionizer</a> and <a href="https://pythonhosted.org/mwsessions/sessionization.html#mwsessions.sessionize">mwsessions.sessionize()</a> can be used by python scripts to group activities into sessions or the command line <a href="https://pythonhosted.org/mwsessions/utilities.html">utilities</a> can be used to operate directly on data files. Such methods have been used to measure editor labor hours.</li>
</ul>
<li><code style="background-color: #f9f9f9; border-radius: 2px; border: 1px solid rgb(221, 221, 221); font-family: monospace, Courier; font-size: 14px; line-height: 22.4px; padding: 1px 4px;">pip install mwpersistence</code><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;"> </span><span style="background-color: white; color: #252525; font-family: sans-serif; font-size: 14px; line-height: 22.4px;">• </span><a class="external text" href="https://github.com/mediawiki-utilities/python-mwpersistence" rel="nofollow" style="color: #663366; font-family: sans-serif; font-size: 14px; line-height: 22.4px; padding-right: 13px; text-decoration: none;">source</a></li>
<ul>
<li>This library provides a set of utilities for measuring content persistence and tracking authorship in MediaWiki revisions.</li>
</ul>
</ul>
<div>
<span style="color: #252525; font-family: sans-serif;"><span style="font-size: 14px; line-height: 22.4px;">And I have a bunch more that are just on the horizon. They represent a sampling of my active research projects.</span></span></div>
</div>
<div>
<ul>
<li>mwmetrics -- User behavioral statistics extraction for MediaWiki editors</li>
<li>mwrefs -- Extract citations, references and scholarly identifiers from MediaWiki</li>
<li>mwevents -- Generalized event extraction and processing framework for MediaWiki</li>
<li>mwtalkpage -- A talk page discussion parser for MediaWiki</li>
</ul>
<div>
It's my goal that researchers who haven't been working with wiki datasets will have a much easier time building off of my work to do their own. I think that a good set of libraries can make a huge difference in this regard. That's my goal. <br />
<br />
I'll be making a more substantial announcement soon. In the meantime, I'm cleaning up and extending documentation and working together some examples that demonstrate how a researcher can compose these small, sharp libraries together to perform powerful analyses of Wikipedia and users in other MediaWiki wikis. Until then, please use these utilities, let me know about bugs and send my your pull requests!</div>
</div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-20369459113507583192015-08-27T11:31:00.003-07:002015-08-27T11:39:00.804-07:00Pre-blog: The life of a traveling technologistSo, I didn't have much time this week and I'm doing this Iron Blogging thing. If you got here looking for a cool discussion of the life of a traveling technologist, I regret to inform you that this will only be a meta-discussion. Once I've completed the proper discussion, I'll put the link right below this paragraph.<br />
<br />
<h2>
What?</h2>
<div>
So my life is pretty weird in a lot of ways. I travel a lot. I generally don't see my teammates at the Wikimedia Foundation for months at a time. Worse, I have a group of friends who are extremely geographically distributed that my geographically local friends don't know about. We only see each other during conferences and other academic events. </div>
<div>
<br /></div>
<div>
Another sort of interesting aspect of my professional life is that I straddle the line between industry and academia. When it comes to the meat of knowledge & knowledge production, there's no conflict. But the timescales are amazingly different. </div>
<div>
<br /></div>
<div>
But through dealing with this, I've worked out some hacks. Some have to do with communication channels and making it feel like you are <i>present </i>even when you are not in the office. Others are my folding bike and the amazing experience I get visiting European cities. </div>
<div>
<br /></div>
<div>
So, I conclude with a promise of future bloggings with photos and insights. I just don't have the time right now!<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://pbs.twimg.com/media/CAO7QtvU8AAPJor.jpg:large" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="180" src="https://pbs.twimg.com/media/CAO7QtvU8AAPJor.jpg:large" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Some of my geographically distributed social network who are <br /><div style="text-align: left;">
<a href="https://pbs.twimg.com/media/CE2anfSUIAAf4WS.jpg:large" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><span style="font-size: 12.8000001907349px;">unknown </span><span style="font-size: 12.8000001907349px;">to my local friends. </span></div>
</td></tr>
</tbody></table>
<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://pbs.twimg.com/media/CE2anfSUIAAf4WS.jpg:large" imageanchor="1" style="clear: left; font-size: 12.8000001907349px; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://pbs.twimg.com/media/CE2anfSUIAAf4WS.jpg:large" width="240" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">My cardboard cutout at the WMF office.</td></tr>
</tbody></table>
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://pbs.twimg.com/media/BygNxfYCUAAdxJf.jpg:large" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="240" src="https://pbs.twimg.com/media/BygNxfYCUAAdxJf.jpg:large" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">My folding bike -- waiting for my flight. </td></tr>
</tbody></table>
<div>
<br /></div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-19263443276302356552015-08-06T08:46:00.002-07:002015-08-06T08:46:56.726-07:00VisualEditor and barriers to entry in WikipediaIn this blog entry, I'm going to briefly cover a presentation that I gave recently as part of the <a href="https://www.mediawiki.org/wiki/Wikimedia_Research/Showcase">Wikimedia Research Showcase</a>. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-qLih9sJthes/VcOAOq0PJFI/AAAAAAAAGcs/zwKJbvAuJjI/s1600/The%2Bnewcomer%2Bgauntlet%2B%2528and%2BVE%2529%2B%25281%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="210" src="http://2.bp.blogspot.com/-qLih9sJthes/VcOAOq0PJFI/AAAAAAAAGcs/zwKJbvAuJjI/s400/The%2Bnewcomer%2Bgauntlet%2B%2528and%2BVE%2529%2B%25281%2529.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><div style="text-align: center;">
<b style="font-size: 12.8000001907349px;">The Newcomer Gauntlet. </b><span style="font-size: 12.8000001907349px;">A t</span><span style="font-size: 12.8000001907349px;">heoretical diagram of barriers to entry in </span></div>
<div style="text-align: center;">
<span style="font-size: 12.8000001907349px;">Wikipedia are depicted </span><span style="font-size: 12.8000001907349px;">with the </span><span style="font-size: 12.8000001907349px;">hypothesized effect of VisualEditor's</span></div>
<div style="text-align: center;">
<span style="font-size: 12.8000001907349px;"> reduction in technical literacy </span><span style="font-size: 12.8000001907349px;">barrier highlighted in green. </span></div>
</td></tr>
</tbody></table>
<br />
<b>TL;DR:</b> We ran an experiment where we gave a <a href="https://en.wikipedia.org/wiki/WYSIWYG">WYSIWYG editor</a> to newly registered editors in Wikipedia and <a href="https://meta.wikimedia.org/wiki/Research:VisualEditor%27s_effect_on_newly_registered_editors">monitored the effect it had on their productivity</a>. We found that <a href="https://meta.wikimedia.org/wiki/Research:VisualEditor%27s_effect_on_newly_registered_editors/May_2015_study#Newcomer_productivity_and_survival">it didn't affect productivity either way</a>. I think that this is because the barriers to entry in Wikipedia primarily consist of social/motivational issues, so reducing the technical literacy barriers that VE targets did not have a meaningful effect. <br />
<i><br /></i>
The talk is embedded below. My talk is first. There's a second talk by some students looking at building a knowledge graph with Wikipedia and some google tech too.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/vGyrVg_qKSM/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/vGyrVg_qKSM?feature=player_embedded" width="320"></iframe></div>
<br />Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-16849287632707208332014-11-30T12:27:00.001-08:002014-11-30T13:12:58.271-08:00Fitting hadoop streaming into my python workflow<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;">I got something working in Hadoop (by reading the documentation[1]) that is going to change the way I work with Wikipedia text data. In order to give you a sense for how excited I am, I'm going to need to talk to you about the way that I approach my work with large datasets. </span><br />
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<h1>
<span style="font-family: Arial, Helvetica, sans-serif;">My python workflow</span></h1>
</div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;">So, I try to do as much work as I can with input/output streams. I'm a bit of a unix geek, so that means I'm thinking about standard-in and standard-out and writing my analytics code as though I were supplementing UNIX core utils. This has nice computational practicalities and it allows you to stream operations together. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<h3>
<span style="font-family: Arial, Helvetica, sans-serif;">Computational practicalities</span></h3>
<span style="font-family: Arial, Helvetica, sans-serif;">The worst bottleneck you're likely to encounter in the course of doing data analysis on large files is memory. Most statistical environments specifically support in-memory analysis of datasets to the exclusion of streaming. This means that, in order to get any work done, you need to first copy all of your data from disk to memory. If your dataset is larger than the available memory, you'll likely end up crashing the machine you are working with -- or at least render it unusable until your process finally crashes.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://upload.wikimedia.org/wikipedia/commons/thumb/f/fb/Streaming_vs._In-memory.svg/1014px-Streaming_vs._In-memory.svg.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Arial, Helvetica, sans-serif;"><img border="0" height="302" src="https://upload.wikimedia.org/wikipedia/commons/thumb/f/fb/Streaming_vs._In-memory.svg/1014px-Streaming_vs._In-memory.svg.png" width="400" /></span></a></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;">On the other hand, python, perl and unix utilities (like cut, shuf, sort, sed, etc.) afford some powerful capability for working with datasets as streams, and therefore, dramatically reducing the memory footprint of a dataset. As the diagram above makes apparent, if you can fit your data operation into a streaming job, then it doesn't matter how big your dataset is and you'll end up needing very little memory. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="color: #222222; font-size: small;">
<h3>
<span style="font-family: Arial, Helvetica, sans-serif;">Streaming operations together</span></h3>
</div>
<div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;">I like Doug McIlroy's summary of the unix philosophy:</span>
</div>
<blockquote class="tr_bq" style="color: #222222; font-size: small;">
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: x-small;">Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.</span></blockquote>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;">By following these principles, you can build "operators" rather than stand-alone programs. This allows you to write less code (which is a Good Thing™) and get more done. Recently, I wrote a paper that required me to wrangle 12 different datasets from 7 different websites -- and being able to integrate my code with unix core utilities was a godsend. Here's a line from my projects Makefile that I think captures the amazingness. </span></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">(echo -e "user_id\tintertime\ttype";<br /> bzcat datasets/originals/aol_search.tsv.bz2 | <br /> tail -n+2 |</span></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">sort -k1,2 |<br /> ./intertimes |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span style="color: #222222;"> sed -r "s/(.*)/\1\tapp view/" | things</span><br /><span style="color: #222222;"> shuf -n 100000) ></span><br /><span style="color: #222222;">datasets/aol_search_intertime.sample.tsv</span></span><br />
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<br />
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;">These lines achieve the following steps of a streaming job:</span></div>
<div>
<ol>
<li><span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: x-small;">Print out new headers: user_id, intertime, type</span></li>
<li><span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: x-small;">Decompress the app_view.tsv.bz2 dataset</span></li>
<li><span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: x-small;">Trim off the original header line</span></li>
<li><span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: x-small;">Sort the dataset by the first two columns (user_id, timestamp)</span></li>
<li><span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: x-small;">Compute the time between events per user (this is my python)</span></li>
<li><span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: x-small;">Add a column of "app view" corresponding to the "type" header</span></li>
<li><span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: x-small;">Randomly sample 100000 observations</span></li>
<li><span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: x-small;">Write it all out to a new data file. </span></li>
</ol>
</div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;">In order to perform this complex operation over 28 million rows, I wrote ~30 lines of python code and put it in the middle of a few unix utilities that do most of the heavy lifting for me. The processes took about 2 minutes and used a couple hundred MB of memory. The random sample that we finally arrive at contains only 100k rows and will happily load into memory in about a second. </span></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="color: #222222; font-size: small;">
<h1>
<span style="font-family: Arial, Helvetica, sans-serif;">Bringing it together with Hadoop streaming</span></h1>
</div>
<div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;">If you've been living under a rock for the last 5 years or so, Hadoop is a framework for performing map/reduce operations that has become the industry standard. Through Hadoop's streaming interface, I can make use of UNIXy streaming utilities to get my python work done faster. Since I'm already thinking about my work as processing data one-row-at-a-time, map/reduce is not that much of a leap. '</span></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;">However, there's one part of my streaming scripts that is very important to me, but I hadn't told you about it yet. When I do behavioral analysis in Wikipedia, I often need to choose a dimension and process user actions in order over time. A good example of actions in Wikipedia is an "edit". There are two really interesting dimensions to process edits: page and user. </span></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; color: #222222; font-size: small; text-align: center;">
<a href="https://upload.wikimedia.org/wikipedia/commons/thumb/f/f0/Data_processing_dimensions.svg/1024px-Data_processing_dimensions.svg.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="188" src="https://upload.wikimedia.org/wikipedia/commons/thumb/f/f0/Data_processing_dimensions.svg/1024px-Data_processing_dimensions.svg.png" width="400" /></a></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;">The diagram above visualizes these two dimensions. If we look at <a href="https://en.wikipedia.org/wiki/Special:Contributions/EpochFail">my edits</a> over time, we are looking at the "user" dimension. If we look at the article <a href="https://en.wikipedia.org/w/index.php?title=Anarchism&action=history">Anarchism</a>'s edits of time, we are looking at the "page" dimension. In order to processed data based on these dimensions, I'll usually <i>sort </i>and <i>partition</i> the data. Notabily, the designers of Wikipedia's XML dumps had this in mind when they put together <a href="https://en.wikipedia.org/wiki/Help:Export">their format</a>. They partition based on page and sort based on timestamp. </span><span style="font-family: Arial, Helvetica, sans-serif;">Yet, hadoop's native map/reduce strategy doesn't afford the ability to insist that I'll see a page's revisions or a user's edits together and in order.</span></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;">Enter <i>secondary sort. </i>Using org.apache.hadoop.mapred.lib.KeyFieldBasedComparator and org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner, I can tell hadoop to sort and partition data before the "reduce" phase of the process. The best part is, these guys are parameterized in the <b>exact same way as UNIX sort</b>! </span><br />
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;">So, let's say you want to process a users activities over time. Set the partitioner divide the data based on the user_id column and tell hadoop to sort based on the revision timestamp. Here's what the call looks like for a dataset I'm working with right now. (first column == user_id, second column == timestamp)</span><br />
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: x-small;"><br /></span>
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;">hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -D mapreduce.output.fileoutputformat.compress=false \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -D mapreduce.input.fileinputformat.split.minsize=300000000 \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -D mapreduce.task.timeout=6000000 \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -D stream.num.map.output.key.fields=2 \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -D mapreduce.partition.keypartitioner.options='-k1,1n' \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -D mapred.output.key.comparator.class="org.apache.hadoop.mapred.lib.KeyFieldBasedComparator" \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -D mapreduce.partition.keycomparator.options='-k1,1n -k2,2n' \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -files intertime.py \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -archives 'hdfs:///user/halfak/hadoopin/virtualenv.zip#virtualenv' \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -input /user/halfak/enwiki_20141025.*.tsv \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -output "/user/halfak/hadoopin/enwiki_intertime/$(date +%Y%m%d%H%M%S)" \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -mapper cat \</span><br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -reducer intertime.py \</span><br />
<br />
<span style="color: #222222; font-family: Courier New, Courier, monospace; font-size: x-small;"> -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner</span></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">So, I'm stoked. This will change the way I do my work.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">1. Which leaves a bit to be desired[2]</span></div>
<div style="color: #222222; font-size: small;">
<span style="font-family: Arial, Helvetica, sans-serif;">2. Finding which error code to <i>google</i> is worse</span></div>
</div>
</div>
</div>
</div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com3tag:blogger.com,1999:blog-1287268094004627280.post-75698465414350007102014-10-17T07:26:00.001-07:002014-10-19T10:44:24.227-07:00Wikipedia's socio-technology as compared to biologyI recently <a href="https://www.youtube.com/watch?v=-We4GZbH3Iw#t=34m05s">gave a presentation at the Wikimedia Foundation's Research Showcase</a> where I compared Wikipedia to a paramecium. I'll be turning that presentation into a blog post later, but in the meantime, I wanted to share a slide from that presentation that I thought was fun. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-DFSvucKzOiQ/VEElanGuB2I/AAAAAAAAFBs/Vg7NTq3Zo3w/s1600/Wikipedia%2Bis%2Bsocio-technical.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-DFSvucKzOiQ/VEElanGuB2I/AAAAAAAAFBs/Vg7NTq3Zo3w/s1600/Wikipedia%2Bis%2Bsocio-technical.png" height="225" width="400" /></a></div>
<br />
From left to right: a small village with about 150 people, the MediaWiki software and extensions, the crowd from Wikimania'14 and Wikipedia. <br />
<br />
Here, I equate organelles to specialized sub-systems of the greater paramecium. In the case of Wikipedia, specialized sub-systems tend to take the form of software. <br />
<br />
See also:<br />
<br />
<ul>
<li>Video: <a href="https://www.youtube.com/watch?v=-We4GZbH3Iw#t=34m05s">https://www.youtube.com/watch?v=-We4GZbH3Iw#t=34m05s</a></li>
<li>Slides: <a href="https://commons.wikimedia.org/wiki/File:Wikipedia_is_socio-technical_--_Research_Showcase_(October,_2014).pdf">https://commons.wikimedia.org/wiki/File:Wikipedia_is_socio-technical_--_Research_Showcase_(October,_2014).pdf</a></li>
</ul>
<div>
<br /></div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0tag:blogger.com,1999:blog-1287268094004627280.post-88620276646633381082014-09-28T09:06:00.001-07:002014-09-28T11:45:04.761-07:00Treading water & touching the lake bottom -- Perspective for young researchers<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Last week, I went to the University of Michigan to visit <a href="http://clifflampe.org/">Cliff Lampe</a> and to speak at the <a href="http://icos.umich.edu/lectures">ICOS lecture series</a>. In order to welcome me to Ann Arbor, Cliff set up a dinner with some of his bright-eyed young grad students. At one point in the conversation, I off-handedly gave some advice on making it through grad school with one’s psyche intact. It got me thinking about the sense of fear I experience when pursuing unknowns.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Getting started on a new project is a disorienting experience. This disorientation can be terrifying and disheartening. However, as a research project gets more mature, the notion of what questions are interesting, how they can be answered, and why those answers matter begin to form fixed points of reference. In my experience, these conceptual footholds bring an intense calming and sense of a productive purpose to a project. They can mean the difference between feeling like one is standing before a soul crushing abyss, and instead, feeling like one is standing on solid ground. I've experienced something like this before grad school and I think it serves as an appropriate metaphor. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">New projects are like being thrown into the middle of a lake at night. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For those of you who did not grow up in Minnesota (a.k.a. “The Land of 10,000 Lakes”) swimming in a lake is an interesting experience about which I’d like to share some insights. I’ve sketched an attractive graphic. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<a href="http://1.bp.blogspot.com/-sIyNUiCrOrM/VChPep0GupI/AAAAAAAAE7g/xpbnEjj0H9k/s1600/Conceptual%2Bdepth%2B(1).png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="http://1.bp.blogspot.com/-sIyNUiCrOrM/VChPep0GupI/AAAAAAAAE7g/xpbnEjj0H9k/s1600/Conceptual%2Bdepth%2B%281%29.png" height="305" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, when you are floating in a lake and you can’t reach the bottom with your feet, you don’t have a good sense for how deep the water is. In the dark water, your imagination of the distance between you and the lake's bottom beneath you can be your enemy. Even if the bottom of the lake is just inches away, it </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">could</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> be a hundred feet away -- and filled with eels, sharks and monsters. This could be seen as a childish way of thinking, but even today I find myself slipping into the spiral of imagined terrors when I swim in large, deep bodies of water. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As you approach the shore, the fear doesn’t progressively subside. Instead, the act of swimming feels a bit like fleeing and that enhances the thoughts of imagined terrors just inches from your feet. Faster swimming leads to more terror enhancement in sneaky spiral that makes you want to swim for your life.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">However, once your feet come into contact with the sandy bottom, the relief is immediate. This knowledge of where the bottom is gives you perspective. You can now measure progress toward the shore. You can stand and stop treading water. There's no room for monsters beneath you anymore. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">OK, back to research projects. I see a lot of young grad students leading a project for the first time struggling in their deep water stage(which is totally normal) and assuming that it’s because they are somehow less competent than others(they aren't) or that their work won't lead to anything useful. This happens all too often when students are making fine progress. They may be getting close to the shore of their research project, but they don't have a sense of any reference points yet. The unknowns still feel infinite and intractable. The direction they're going could be leading them to deeper water. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So far as I can tell, everyone experiences the deep water stage of a research project, no matter how senior. Seasoned researchers have made peace with, or even celebrate, this disorienting part of research in the same way that veterans of the lake have made peace with their imagined deep-water monsters. The trick is to staying sane is to recognize that sneaky spiral of fear when it happens.</span></div>
Aaron Halfakerhttp://www.blogger.com/profile/15426808888622986879noreply@blogger.com0