Summarizing Data

{"url": "https://mdogucu.ics.uci.edu/teaching/stats67-fa19/slide/slide-2a-summarize-data.html", "content": "\r\n\r\n \r\n Summarizing Data\r\n \r\n \r\n \r\n \r\n \r\n \r\n

\r\nclass: center, middle, inverse, title-slide\r\n\r\n# Summarizing Data\r\n### Dr.\u00a0Dogucu\r\n### 2019-10-07\r\n\r\n---\r\n\r\n\r\nlayout: true\r\n  \r\n<div class=\"my-header\"></div>\r\n\r\n<div class=\"my-footer\"> \r\n Copyright &copy; <a href=\"https://mdogucu.ics.uci.edu\">Dr. Mine Dogucu</a>. All Rights Reserved.</div> \r\n\r\n---\r\n\r\n\r\n\r\n## Check\r\n\r\n- Font size\r\n---\r\n## Announcements\r\n\r\n- Additional office hours: Thursdays 3 - 4 pm.\r\n\r\n- [OpenIntro has videos](https://www.youtube.com/user/OpenIntroOrg/playlists)\r\n- Do not trust Canvas for your grades. Grading policies are on the course website. \r\n\r\n- Even though I respect your preferred name, Gradescope can only detect your official name that is on the Course roster. \r\n\r\n---\r\n\r\n## Clarifications\r\n\r\n- Reading questions are due BEFORE class. You only get ONE attempt and it is timed. \r\n- Week 1 was for getting used to the course. Make sure you read instructions.\r\n- Lab assignments are to be completed in class during discussion sessions.\r\n\r\n---\r\n## Clarifications\r\n\r\n- Reading Question  regarding music study skills\r\n\r\n---\r\n\r\n## Attendance\r\n\r\nIf you do not attend lecture or discussion, you will be lost. \r\n\r\nThis is NOT a class that you can only show up to midterm and final, and pass.\r\n\r\n---\r\n## All About You Survey\r\n\r\n- Thank you. I am happy to get to know you.\r\n- Worry: Programming, R, Probability, Integrals, Failing, Grades, No AP Stats, Group work, Staying awake during lecture, Learning things I may not ever use, Laptop issues, Time management, Course load, Catching Up after late registeration, Wednesday schedule,  Reading, New concepts/vocabulary, not CS major, no graphing calculator, Waking up/ driving, Nothing \r\n---\r\n## RStudio Cloud\r\n\r\n<img src = \"img/cloud-options.png\" class=\"center-image\">  </img>\r\n---\r\n## Why R?\r\n\r\n- First and foremost you will add one more language to your skill set. You should know both Python and R.\r\n- Unlike Python, which is a general purpose language, R was built for data handling and analysis and is a powerful tool for data science. \r\n- R is rich and provides functions for any typical statistical model that you can think of.\r\n- R provides a comprehensive data visualization tools that are easy to use such as `ggplot`.\r\n\r\n---\r\n\r\n## Review - Study Design\r\n\r\nPatients with a certain disease at a state hospital participate in a research study. They are randomly split into two and either receive a new found drug or a placebo based on their group. Researchers observe that the patients who have taken the drug have been cured from the disease. Which of the following is true? \r\n\r\na) Researchers can conclude that this drug cures the disease.  \r\nb) This is an observational study since it says \"observe\". Researchers cannot conclude that this drug cures the disease.   \r\nc) Researchers can conclude that this drug cures the disease only for this sample. \r\n\r\n---\r\n## Review - Study Design\r\n\r\nPatients with a certain disease at a state hospital participate in a research study. They are randomly split into two and either receive a new found drug or a placebo based on their group. Researchers observe that the patients who have taken the drug have been cured from the disease. Which of the following is true? \r\n\r\na) Researchers can conclude that this drug cures the disease.  \r\nb) This is an observational study since it says \"observe\". Researchers cannot conclude that this drug cures the disease.   \r\n__c) Researchers can conclude that this drug cures the disease only for this sample.__ \r\n\r\n---\r\n## A/B testing\r\n\r\n- A/B testing is basically an experiment that compares two versions (A and B) of a single variable.\r\n- It is commonly used on measuring online activies such as click-through rates (usually online ads) and revenue per user.\r\n- Example: Let's assume that on your company's web page you show the ads on the right side. You want to test whether moving the ad to the header will increase clicks on the ad (and money for your company). For randomly selected half of the users, you can display ads on the right (version A - control), and for the other half you can display the ads on the header (version B - treatment) and compare. \r\n\r\n---\r\n## Review - Types of Variables\r\n\r\n\r\n```r\r\ntitanic_train %>% \r\n  select(PassengerId, Name) %>% \r\n  glimpse()\r\n```\r\n\r\n```\r\n## Observations: 891\r\n## Variables: 2\r\n## $ PassengerId <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,...\r\n## $ Name        <chr> \"Braund, Mr. Owen Harris\", \"Cumings, Mrs. John Bra...\r\n```\r\n\r\nNote that both `PassengerId` and `Name` are nomimal variables. There is really no meaningful ordering or grouping for these variables. \r\n---\r\n## Review - Study Design\r\nMohammed claims that he can tell the difference between tap water and bottled water. His friend, Phoebe does not believe him. They want to test his claim. They invite their friend Xuan to help out. Xuan pours bottled or tap water into a cup and gives it to Phoebe who then gives this to Mohammed. They repeat this twenty times. The study design is\r\n\r\na. a confounding design.  \r\nb. a single-blind design.  \r\nc. a double-blind design.  \r\n---\r\n## Review - Study Design\r\nMohammed claims that he can tell the difference between tap water and bottled water. His friend, Phoebe does not believe him. They want to test his claim. They invite their friend Xuan to help out. Xuan pours bottled or tap water into a cup and gives it to Phoebe who then gives this to Mohammed. They repeat this twenty times. The study design is\r\n\r\na. a confounding design.  \r\nb. a single-blind design.  \r\nc. __a double-blind design__.  \r\n---\r\n## Review - Relationships of Variables\r\n\r\nIf two variables are related/associated they are called __dependent__ variables (e.g. having a college degree and income)\r\n\r\nIf two variables are not related/associated they are called __independent__ variables (e.g. your hair color and whether it will rain in Gaborone.)\r\n---\r\n## Some R terminology \r\n\r\n\r\n```r\r\nglimpse(titanic_train)\r\n```\r\n\r\n```\r\n## Observations: 891\r\n## Variables: 12\r\n## $ PassengerId <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,...\r\n## $ Survived    <int> 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0,...\r\n## $ Pclass      <int> 3, 1, 3, 1, 3, 3, 1, 3, 3, 2, 3, 1, 3, 3, 3, 2, 3,...\r\n## $ Name        <chr> \"Braund, Mr. Owen Harris\", \"Cumings, Mrs. John Bra...\r\n## $ Sex         <chr> \"male\", \"female\", \"female\", \"female\", \"male\", \"mal...\r\n## $ Age         <dbl> 22, 38, 26, 35, 35, NA, 54, 2, 27, 14, 4, 58, 20, ...\r\n## $ SibSp       <int> 1, 1, 0, 1, 0, 0, 0, 3, 0, 1, 1, 0, 0, 1, 0, 0, 4,...\r\n## $ Parch       <int> 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 1, 0, 0, 5, 0, 0, 1,...\r\n## $ Ticket      <chr> \"A/5 21171\", \"PC 17599\", \"STON/O2. 3101282\", \"1138...\r\n## $ Fare        <dbl> 7.2500, 71.2833, 7.9250, 53.1000, 8.0500, 8.4583, ...\r\n## $ Cabin       <chr> \"\", \"C85\", \"\", \"C123\", \"\", \"\", \"E46\", \"\", \"\", \"\", ...\r\n## $ Embarked    <chr> \"S\", \"C\", \"S\", \"S\", \"S\", \"Q\", \"S\", \"S\", \"S\", \"C\", ...\r\n```\r\n---\r\n## Some R terminology\r\n\r\nR is a functional language. In this case `glimpse()` is a function and `titanic_train` is the argument. \r\n\r\n---\r\n## Review - __%>%__ (aka pipe operator)\r\n\r\nSince R is functional let's remember functions from algebra. If you want to execute fog(x), you have three choices in R\r\n\r\n\r\n```r\r\nf(g(x))\r\n```\r\n\r\n\r\n\r\n```r\r\ng(x) %>% \r\n  f()\r\n```\r\n\r\n\r\n```r\r\nx %>% \r\n  g() %>% \r\n  f()\r\n```\r\n\r\n---\r\n## Review - __%>%__ (aka pipe operator)\r\n\r\n\r\n\r\n```r\r\nglimpse(titanic_train)\r\n```\r\n\r\n\r\n```r\r\ntitanic_train %>% \r\n  glimpse()\r\n```\r\n\r\n---\r\n\r\n## Review - __%>%__ (aka pipe operator)\r\n\r\n\r\n```r\r\ncount(titanic_train, Survived)\r\n```\r\n\r\n```\r\n## # A tibble: 2 x 2\r\n##   Survived     n\r\n##      <int> <int>\r\n## 1        0   549\r\n## 2        1   342\r\n```\r\n\r\n\r\n```r\r\ntitanic_train %>% \r\n  count(Survived)\r\n```\r\n\r\n```\r\n## # A tibble: 2 x 2\r\n##   Survived     n\r\n##      <int> <int>\r\n## 1        0   549\r\n## 2        1   342\r\n```\r\n\r\n---\r\n\r\n## Review - `count()`\r\n\r\nCount can take more than two variables.\r\n\r\n\r\n```r\r\ncount(titanic_train, Survived, Pclass)\r\n```\r\n\r\n```\r\n## # A tibble: 6 x 3\r\n##   Survived Pclass     n\r\n##      <int>  <int> <int>\r\n## 1        0      1    80\r\n## 2        0      2    97\r\n## 3        0      3   372\r\n## 4        1      1   136\r\n## 5        1      2    87\r\n## 6        1      3   119\r\n```\r\n\r\n---\r\n## Review - Multiple Lines\r\n\r\nYou can break code over multiple lines\r\n\r\n\r\n```r\r\ncount(titanic_train, \r\n      Survived, \r\n      Pclass)\r\n```\r\n\r\n```\r\n## # A tibble: 6 x 3\r\n##   Survived Pclass     n\r\n##      <int>  <int> <int>\r\n## 1        0      1    80\r\n## 2        0      2    97\r\n## 3        0      3   372\r\n## 4        1      1   136\r\n## 5        1      2    87\r\n## 6        1      3   119\r\n```\r\n---\r\n## Summarizing Sleep Data by Hand\r\n\r\n\r\n```\r\n##  [1] \"7\"                        \"8\"                       \r\n##  [3] \"8\"                        \"8\"                       \r\n##  [5] \"8\"                        \"7\"                       \r\n##  [7] \"8\"                        \"4\"                       \r\n##  [9] \"7 (very rare occurrence)\" \"7.5\"\r\n```\r\n\r\n\r\n\r\nCalculate 1)mean, 2) median, 3) mode, 4) first quartile, 5) third quartile, 6) interquartile range and \r\n\r\nDraw a boxplot\r\n---\r\n## Summarizing Data\r\n\r\n\r\n\r\n```r\r\ntitanic_train %>% \r\n  summarize(mean(Fare), median(Fare),\r\n            sd(Fare), var(Fare), \r\n            min(Fare), max(Fare))\r\n```\r\n\r\n```\r\n##   mean(Fare) median(Fare) sd(Fare) var(Fare) min(Fare) max(Fare)\r\n## 1   32.20421      14.4542 49.69343  2469.437         0  512.3292\r\n```\r\n---\r\n## Summarizing Data - Quantiles \r\n\r\n\r\n```r\r\ntitanic_train %>% \r\n  summarize(median(Fare),\r\n            quantile(Fare, 0.50))\r\n```\r\n\r\n```\r\n##   median(Fare) quantile(Fare, 0.5)\r\n## 1      14.4542             14.4542\r\n```\r\n---\r\n\r\n## Summarizing Data - Quantiles \r\n\r\n\r\n```r\r\ntitanic_train %>% \r\n  summarize(q1 = quantile(Fare, 0.25),\r\n            q3 = quantile(Fare, 0.75))\r\n```\r\n\r\n```\r\n##       q1 q3\r\n## 1 7.9104 31\r\n```\r\n---\r\n\r\n## Summarizing Data - `group_by()`\r\n\r\n\r\n```r\r\ntitanic_train %>% \r\n  group_by(Pclass) %>% \r\n  summarize(mean_fare = mean(Fare))\r\n```\r\n\r\n```\r\n## # A tibble: 3 x 2\r\n##   Pclass mean_fare\r\n##    <int>     <dbl>\r\n## 1      1      84.2\r\n## 2      2      20.7\r\n## 3      3      13.7\r\n```\r\n\r\nYou can only group by a categorical variable.\r\nImagine doing this by hand. You would first take the `Fare` amounts and then divide them into three groups based on `Pclass` and then calculate the mean for each group. \r\n\r\n---\r\n## Effect of Outliers\r\n\r\nLet's assume that the height in our classroom has bell-shaped symmetric distribution. In such distributions, mean = median = mode. \r\n\r\nIf Michael Jordan (6 feet 6 inches - 198.12 cm) were to walk into our classroom. How would the distibritution of our heights change? Describe the shape and spread. Order the mean, median, and mode in descending order.\r\n\r\nIf Tyrion Lannister (4 feet 5 inches - 135.00 cm) were to walk into our classroom. How would the distibritution of our heights change? Describe the shape and spread. Order the mean, median, and mode in descending order.\r\n\r\n---\r\n## Plots, plots, and more plots\r\n\r\n- for a single categorical variable\r\n- for a pair of categorical variables\r\n- for a single continuous variable\r\n- for a pair of continuous variables\r\n- for a single categorical and a single continuous variables\r\n\r\n\r\n---\r\n\r\n## Within One Week\r\n\r\n* Due Wednesday:  \r\n  + Reading and Reading Questions \r\n* Due Friday:  \r\n + HW  (Introduction to Data) \r\n Note that your new homework will be posted on this      Friday. Make sure to check the course website.\r\n* Due Monday:   \r\n  + Reading and Reading Questions\r\n

\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n \r\n\r\n", "encoding": "utf-8"}