Search This Blog

Week 20: Starting Paper and Fixing Frontend

For last week I was in charge of preparing my group to start the user study and writing our research paper.

For the user study, following the IRB form questions, I completed a google forms (so that we could start surveying as soon as we were done with the meeting with the professor). Furthermore, I started working on writing out an outline.

Single Spaces: about 9 - 10 pages each

Abstract

Executive Summary

Introduction

Ranking

Definition

Rankings enable the proceassing of large multifaceted data to synthesize a representation of reality, allowing individuals to weigh and observe in clearer detail the choices they are faced with.

Importance

Employed in everyday life and decision making: website search, picking out a place to eat, deciding on the best college

Rankings are based on certain attributes. Each attribute can have a higher weight/contribution to the overall ranking. Available online rankings may hide which attribute are considered, fuzzing the meaning of the rankings.

It is important to be wary of rankings without disclosed weights and their attributes. To deflect this problem, it is vital to encourage individuals to formulate their own rankings.

Machine learning techniques

Manually ranking (assigning each attribute a specific weight) can be a difficult task to the individual who is unfamiliar with the whole dataset or is unsure about the importance of each attribute. Coming up with the weights requires prior knowledge-- not friendly to beginners

Machine learning techniques allow for automatic rankings of datasets.

RANKit

RANKit is an online ranking application that provides solutions for analyzing and exploring rankings. The system uses machine learning to automatically construct rankings based on partial input from users. Intuitive interfaces allow for the effective building, exploring, and explaining of rankings.

Background

Motivation

Problems in ranking

Misleading results

Case study

Needs to be a framework that helps users get an intuition for how rankings work.

Fairness in ranking

Groups = equal representation - statistical parity

Equalized odds, if you train a model. The amount of errors you make in predicting model - should be the same for each group

The ranking you assign should mean something

Intuitiveness of UI

Clean and simple design

Small amount of text

Unobtrusive buttons

Colors

Color blindness

Common practices in most websites

Tabs

Buttons

Hover over states

Cursor changing

Related Work

Matters

Description of tool

Observations - key features

How it inspired RANKit

Lineup

Description of tool

Observations - key features

How it inspired RANKit

Podium

Description of tool

Observations - key features

Methodology

Goals

Encourage critical thinking and spread awareness in ranking (teach more about how rankings are formulated)

Create an unbiased and easy to use tool for individuals both knowledgeable and not in the subject of interest

Overview

Deciding a platform type

Web application

Desktop application

Mobile application

Researching languages

Javascript

Why would it be useful?

What are the drawbacks?

Python

Why would it be useful?

What are the drawbacks?

Hybrid of Python and Javascript

Why would it be useful?

What are the drawbacks?

Design and Implementation

Splitting the backend and frontend

The Server

Machine learning algorithm to determine rankings

Importance of performance

The Client

Capture interest with a landing page

Machine learning tool with a friendly UI

Visualization of final ranking

Interview

Goals

Evaluate which method of building a ranking is most favorable among the three presented

Estimate an amount of partial information the user is willing to input

Determine the intuitiveness of the user interface

Procedure

The method for gathering data will involve in-person interviews.

The interviews will consist of a one-time session where the participants will be asked to perform two tasks using our ranking application. After performing each task, they will be asked to rate their overall quality of the interaction using a questionnaire.

Questions

How can we test the goals?

Online Survey

Goals

Evaluate which method of building a ranking is most favorably rated among the three Build views

Procedure

The method for gathering data will involve an online questionnaire participation.

The interviews will consist of a one-time session where the participants will be asked to perform six tasks using the ranking application. After performing each task, they will be asked to rate their overall quality of the interaction using a questionnaire.

Questions

How can we test the goals?

Result

Goals

What we’ve done to address the below goals

Encourage critical thinking and spread awareness in ranking (teach more about how rankings are formulated)

Create an unbiased and easy to use tool for individuals both knowledgeable and not in the subject of interest

Overview

Backend language: Python

Description

Why we chose it

Frontend language: Template Jinja2, JS

Description

Why we chose it

Design and Implementation

The Server

Architecture of the system

File structure

Blueprint archetype

The ranking script

The Client

Landing page

Different iterations

Button placement

Build Methods

Visual description

When this can be useful

Algorithm that generates pairwise

Explore

Widget that displays weights of each attribute (that determined the final ranking)

Table features

Interview

Intuitiveness of the Interface

Comparing Ranking Techniques

Feedback

Survey

Data and Analysis

Overview of the questions and what they are trying to test

Data

Analysis

Future Works

Studies

Features

Rank by attribute

Rank by multiple attributes

More interactibility in Explore

Conclusions (1 page)

Appendices

Survey Questions

Interview Questions

Get link

Facebook

Twitter

Pinterest

Email

Other Apps

Comments

Post a Comment

Popular posts from this blog

With the Rankit paper submitted, it was time for me to change gears and dive into the research with MaryAnn.

For me, this week revolved around getting up to speed with the fair ranking research. I read over the current in progress fair ranking paper and attended meetings where MaryAnn and Caitlin helped familiarize me with the code base and research that they've completed.

After formulating an experimental plan, we needed to have some datasets to run those experiments on. There are some requirements that classify a good dataset for this task: it must be non-trivial in size, needs to include a protective attribute (such as gender or race), and it must have some true ranking. While it is possible to find a datasets that satisfies two out of the three requirements, it becomes difficult to satisfy all of the requirements.

Encompassing a dataset from Rankit, list of Fifa 2018 players was added as a possibility. It contained 17981 players, has a protective attribute (age and nationality), and potentially has a true ranking. The true ranking can be based on goals scored, or money earned by the player. Some ranking data can also be found over this datasets, although it does not encompass all of the players.

Hospitals and doctors also have a significant amount of data, both attribute and entity-vise. However, finding a true ranking might prove to be impossible.

Taking into account the feedback received from the our mentors, we updated the section analyzing the outcome of the online user study. We updated the machine learning section to include more references and added more charts to the whole paper.
The team also discussed the next steps and observed new features to be implemented.