Showing posts with label qualitative. Show all posts
Showing posts with label qualitative. Show all posts

Thursday, October 20, 2016

Best practices for AI in the social spaces: Integrated refutations

So, I was listening to an NPR show titled "Digging into Facebook's File on You".  At some point, there was some casual discussion of laws that some countries in the European Union have re. users' ability to review and correct mistakes in data that is stored about them.  This made me realize that ORES needs a good mechanism for you to review how the system classifies you and your stuff. 

As soon as I realized this, I wrote up a ticket describing my rough thoughts on what a "refutation" system would look like.  From https://phabricator.wikimedia.org/T148700:
Build a user-friendly UI for reviewing how ORES has classified you or any query-able slice of activity.
Include UI & API to refute classifications endpoints (via OAuth).
Include UI to curate "refutations" (suppress & meta-review).
Notes:
  • Refutations need freeform text. Freeform text *must* have a suppression system.
  • We should include refutations in ORES query results.
  • We can borrow from MediaWiki's global user rights (ORES is global) for curation privs.
On one hand, I'm excited by this idea because I think it will be a very interesting exploration into what kinds of feedback people want to give to ORES' classifications.  I think it will provide a good means for humans to disagree with the machine in ways that are effective.  It's important that we don't hide false-positive/negative reports somewhere that no one will ever see.  It would be better if such reports were available as part of a query result -- so that ORES decision can be flagged and directly challenged by a human.

On the other hand, I'm a little bit embarrassed that this wasn't part of the plan all along.  I guess in a way, I expected the wiki to fill this infrastructure -- of critiquing ORES.  But critiques on the wiki are hidden from the API that tools use to consume score and relegating them to a less effectual class of data.  I feel a little bit stupid that it never occurred to me that humans should be able to affect ORES' output.

Now to find the resources to construct this Meta-ORES system...

Sunday, March 13, 2016

Brief discussion: The Effects and Antecedents of Conflict in FLOSS dev.

Hey folks.  For this weeks blogging, I'll be reading and thinking about:
A Filippova, H ChoThe Effects and Antecedents of Conflict in Free and Open Source Software DevelopmentCSCW, 2016
This isn't a review or really a summary -- just some thoughts I had that I want to share.

Transformational leadership

See https://en.wikipedia.org/wiki/Transformational_leadership
Filippova et al. highlight a "transactional leadership" style as one of the factors mitigating the negative effects of high-conflict FOSS teams.  

I don't really like the word "transformational" since, to me, it seems to fail to highlight the key meaning of the term -- collaboration.  A transactional leader doesn't own the process of change, but rather works with others to put together a vision and enact a change with other committed individuals.  It involves convincing other of a directional change or new effort rather than enforcing it through force or punishment/reward.  I think the key insight that transactional leaders have is that they do not own the process of change.  At most they can be a key contributor to it.  There will need to be a conversation and some sort of consensus before real change can happen.  In my practice as a leader of a small group of volunteers developing a FOSS project, I can't imagine operating any other way.  I must lead by example -- by making a convincing case for what we should do and allowing myself to be convinced by other.  The direction we set will be owned by all.  I need my team to feel this ownership and shared identity.  This is a trick when there are deadlines and a disagreement threatens delaying substantial work, but in my experience, deadlines aren't that dead and delays can be real opportunities to step back and ask why the disagreement exists in the first place.  

In the end, if I can't convince you (gentle reader, my assumed collaborator) that my idea is good and worth pursuing then maybe it's not actually good or worth pursuing.  In a way, I see transactional leadership style a lot like code review.  If I want to reprogram our team structure/plans, then I should be able to get the changes reviewed and supported by others.  In the process of review, we can increase our shared notion of norms/goals and make sure that the implemented changes are actually good!

Different types of conflict

So, I've been studying conflict patterns in Wikipedia for a while, but I've never really dug into the literature about different types of conflict.  Of course, it's obvious that there *are* different types.  I've written about this in the past around reverts in Wikipedia, but it's much more useful to apply past through on the subject than to inject my own naive point of view.  Luckily, Filippova et al. provide a nice summary of Task conflict, Affective conflict, Process conflict, and Normative conflict.

  • Task conflict: Conflict about what needs to be done.  E.g. do we engineer ORES to use celery workers or do we just plan to have a large pool of independent uwsgi threads?  While this can be good in that it brings a diverse perspective of possible implementations, it also might turn into a religious battle over the Right Way To Do Things(TM).
  • Affective conflict: AKA drama.  Conflict due to bad blood -- relationships between people might cause conflict regardless of any task disagreement.   E.g. do we attribute the failure of a team member to complete a task to the complexity of the task or to their general incompetence? 
  • Process conflict: Disagreements over how to do tasks.  E.g. do we require code-review and non-self-merges for *everything* or are there some reasonable exceptions.  On the Revision Scoring project, I generally make process declarations that go unchallenged and then we iterate whenever the process seems to be not working.  So far, I wouldn't really say that this has escalated to "conflict" yet, but I could see how it might.  
  • Normative conflict: Disagreements about "group function".  E.g. do we generally pursue a caution-first strategy or an open-first strategy (everything is open until a problem arises vs. everything is closed until we know we can safely open it)?  This is a discussion that I'd like to come back to in a future blog post as I'm very opinionated about how the norms of Wikipedia (bold inclusionism & openness) should be extended to the software development community around MediaWiki.