1️⃣ AI•DS/πŸ“• λ¨Έμ‹ λŸ¬λ‹

λ°μ΄ν„°λ§ˆμ΄λ‹ Preprocessing β‘‘

isdawell 2023. 3. 15. 12:31
728x90

 

1. Types of data sets 

 

 

 

•  Relational Records 

β†ͺ collections of records, each of which consists of a fixed set of attributes 

 

 

 

 

•  Data matrix: record data 

β†ͺ If the data objects have the same fixed set of numeric attributes, the data objects can be thought of as points in a multi-dimensional space, where each dimension represents a distinct attribute 

β†ͺ m by n matrix, m rows, n columns

 

 

 

 

•  Document data: record data 

β†ͺ Each document becomes a term vector (word vector) 

β†ͺ Each term is a component of the vector 

β†ͺ The value of each component is the # of times the corresponding term occurs in the document 

 

 

 

 

•  Transaction data: record data 

β†ͺ special type of record data, where each record involves a set of items 

β†ͺ Also called market basket data 

 

 

 

 

•  WWW data: graph data 

β†ͺ world wide web: HTML links between web pages can be represented as a graph 

 

 

 

•  Social network data: graph data 

β†ͺ Relationships (friends, followers) between users can be modeled as a graph 

 

 

 

•  Molecular structures: graph data 

β†ͺ Chemical compounds originally have the graph structure 

 

 

 

•  Sequential data:  ordered data 

β†ͺ Sequences of transaction 

 

 

 

•  Time series data:  ordered data 

β†ͺ Sequences of data points, measured typically at successive items spaced at uniform time intervals 

(일, μ›”, 연도) 

 

 

 

•  Genetic Sequence data:  ordered data 

β†ͺ Nucleic acids consist of a chain of linked units called nucleotides 

 

 

 

•  Trajectory data:  ordered data 

β†ͺ Sequences of the locations and timestamps of moving objects  →  Usually represented by latitude and longitude 

 

 

 

 

•  Text data:  ordered data 

 

 

 

 

 

2. Data objects and attribute types 

 

•  Data objects 

β†ͺ data set can often be viewed as a collection of data objects 

β†ͺ data object (or sample, example, instance, point, tuple) represents an entry 

 

 

β†ͺ data objects are described by attributes 

β†ͺ In relational databases, rows are data objects, columns are attributes 

 

 

•  Attributes

β†ͺ attribute (or dimension, feature) is a property or characteristic of an object 

β†ͺ Types 

    β–Έ categorical : qualitative → Nominal, Ordinal 

    β–Έ Numeric : quantitative → Interval, Ratio 

 

 

 

β‘   Categorical attributes types

 

1) Nominal : categories, states, or "names of things" 

 

 

 

2) Binary : special type of nominal attribute

β–Έ Nominaattributete with only 2 states (0,1) 

β–Έ Symmetric binary : both outcomes s equally important 

    ⇨ ex. female and male 

β–Έ Asymmetric binary : outcomes not equally important 

    ⇨ ex. medical test (positive vs negative : positive is more important 

β–Έ Convention : assigning to 1 most important outcome 

    ⇨ ex. HIV positive 

 

 

3) Ordinal : values have a meaningful order (ranking), but the magnitude between successive values is not known 

β–Έ Ex. Sizes (={small, medium, large}), grades, army rankings 

 

β–Έ CQA ) the median of even-numbered ordinal dataset cannot be accurately calculated, but ex, 9 shirts can have sizes from {XXS, XS, S, M, L, XL} in ascending order as follows : XXS, XS, S, S, M, M, L, XL. Then the median is clearly S.

β–Έ CQA ) we do not define the order for any set of attributes. For example, the lexicographic order for names is just for convenience.

 

 

 

β‘‘  Numeric attribute types

 

1) Interval 

β–Έ Values are measured on a scale of equal-sized units: values have order 

   β‡¨ ex . temperature in C∘ or  F∘, calendar dates 

β–Έ No true zero-point 

   β‡¨ ex. 0C∘ is in a physical sense, arbitrary (cf. 0K∘) ⇨  In a physical sense, zero celsius is arbitrary

 

2) Ratio 

β–Έ Inherent zero point (true zero point) 

β–Έ We can speak of values as being an order of magnitude larger than the units of measurement 

   β‡¨ ex. 10K∘ is twice as high as 5K∘ 

   β‡¨ ex. temperature in Kelvin, length, counts, monetary quantities 

 

 

β‘’ Possible data analysis

 

 

⇨ Ordinal attributes λŠ” 쀑앙값, percentile 을 계산할 수 μžˆλ‹€. (CQA μ°Έκ³ ) 

⇨ interval attributes λŠ” ratio λ‚˜ coefficient of variation 을 계산할 수 μ—†λ‹€. 

 

 

 

•  Continuous vs Discrete Attributes

β†ͺ Discrete attribute: A finite number of possible values, can be represented as integer variables 

β†ͺ Continuous attribute: infinite number (ex. temperature, height, weight), Typically represented as floating-point variables 

 

 

 

 

 

 

 

3. Basic statistical description of data 

 

•  Central tendency 

β†ͺ Mean 

β†ͺ Median : The middle value of the ordered set if the number of data is odd; otherwise, the average of the two middle values (데이터가 ν™€μˆ˜κ°œλ©΄ κ°€μš΄λ° κ°’, 짝수개면 쀑간에 μžˆλŠ” 두 κ°’μ˜ 평균값) 

β†ͺ Mode : The value that occurs most frequently in the set 

    ⇨ Unimodal : only one mode 

    ⇨ Bimodal : two modes 

    ⇨ Multimodal : multiple modes 

 

 

 

 

•  Types of Measures 

 

β†ͺ Distributive: A measure that can be computed for a given data set by partitioning the data into smaller subsets, computing the measure for each subset, and then merging the results in order to arrive at the measure’s value for the original (entire) data set 

  ⇨ ex. sum, count, min, max 

 

β†ͺ Algebraic: A measure that can be computed by applying an algebraic function to one or more distributive measures. 

  ⇨ ex. mean = sum/count 

 

 

 

β†ͺ Holistic: A measure that must be computed on the entire data set as a whole

  ⇨ ex. median → exact value for the whole dataset,cannot be calculated by each subset

 

 

 

 

 

•  Approximation of the Median 

 

β†ͺ The median value of a data set can be easily approximated by interpolation for grouped data 

 

 

 

•  Symmetric vs Skewed data 

 

 

⇨ Positively skewed (long right tail) : mode < median < mean 

⇨ Negatively skewed (long left tail) : mean < median < mode 

 

 

 

 

•  Dispersion of data 

β†ͺ Quartiles 

    ⇨  Q1 (25th percentile) , Q3 (75th percentile), Q2 (median, 50th percentile) 

    ⇨  IQR = Q3 - Q1 

    ⇨  Five number summary : min, Q1,Q2,Q3, max

    ⇨  Outlier : usually , a value higher/lower than 1.5 X IQR 

 

β†ͺ Variance and standard deviation 

 

 

 

•  Quartile κ΅¬ν•˜λŠ” 방법 

β†ͺ μ²«λ²ˆμ§Έ 방법 

    ⇨  쀑앙값을 ν¬ν•¨ν•˜μ§€ μ•Šκ³ , 쀑앙값을 κΈ°μ€€μœΌλ‘œ 절반으둜 데이터λ₯Ό λ‚˜λˆ„μ–΄μ„œ κ³„μ‚°ν•˜λŠ” 방법 

β†ͺ λ‘λ²ˆμ§Έ 방법 

    ⇨  데이터가 ν™€μˆ˜κ°œμ—¬μ„œ 쀑앙값이 데이터 λ‚΄μ—μ„œ μ°Ύμ•„μ§ˆ λ•Œ (datum 이라 뢀름) λŠ” 쀑앙값을 ν¬ν•¨μ‹œν‚€κ³  절반으둜 λ‚˜λˆ„μ–΄ κ³„μ‚°ν•œλ‹€. λ§Œμ•½, datum 이 μ•„λ‹ˆλΌλ©΄, 첫번째 λ°©μ‹μœΌλ‘œ (쀑앙값을 ν¬ν•¨ν•˜μ§€ μ•ŠμŒ) κ³„μ‚°ν•œλ‹€. 

 

 

 

 

 

 

•  Graphic Displays of Basic Statistical Descriptions

 

β‘  box plot 

 

 

 

 

 

β‘‘ histogram ⭐ 

 

β–Έ showing a visual impression of the distribution of data 

β–Έ νžˆμŠ€ν† κ·Έλž¨ μ§μ‚¬κ°ν˜• 높이 = frequency density of the interval : frequency/width of the interval 

β–Έ The total area is equal to the number of data 

 

 

β‘’ Quantile plot 

 

β–Έ For a value xi, when data are sorted in increasing order, fi, indicates that approximately 100 fi % of the data are below or equal to xi 

 

 

β–Έ kth q-quantile 

   β‡¨ 1 <= k <= (q-1) 

   β‡¨ q = 4 : quartile, 1st-4quantile is Q1, 2nd-4quantile is Q2, 1rd-4quantile is Q3

   β‡¨ q = 100 : percentile 

 

 

 

β‘£ QQ plot

 

β–Έ comparing two probability distributions by plotting their quantiles against each other. 

 

β–Έ If the two distributions being compared are identical, the Q-Q plot follows the line y = x

 

 

β‘€ Scatter plot 

 

β–Έ  A type of mathematical diagram using Cartesian coordinates to display values for two variables for a set of data

β–Έ QQ plot 은 x와 y좕에 ν•΄λ‹Ήν•˜λŠ” λ‹¨μœ„κ°’μ΄ λ™μΌν•œ 반면 (ex. # of item sold in branch1, # of item sold in branch2) scatter plot 은 λ‹¨μœ„κ°€ λ‹€λ₯Έ 두 λ³€μˆ˜κ°€ 와도 상관 μ—†μŒ

 

 

 

 

 

 

4. Data visualization techniques 

 

•  Thematic Cartography

 

β‘  Choropleth

 

 

β‘‘ Proportional symbol

 

 

 

β‘’ Isarithmic

 

 

 

 

β‘£ Dot

 

 

 

 

β‘€ Dasymetric

 

 

 

⇨ similar to choropleth map, but one difference is that the regions are “not predefined”

 

 

 

 

5. Data similarity and dissimilarity 

 

•  Dissimilarity between Nominal attributes 

 

 

 

 

test - 1 에 λŒ€ν•œ attribute 의 λΉ„μœ μ‚¬μ„± 계산 

 

 

 

 

•  Dissimilarity between Binary attributes

 

 

β–Έ symmetric binary variable 

 

λΆ„μžλŠ” mismatch 의 개수

 

β–Έ asymmetric binary variable 

 

λΆ„μžλŠ” mismatch의 개수

 

⇨ asymmetric 은 두 개의 μƒνƒœκ°€ λ™μΌν•œ μ€‘μš”μ„±μ„ κ°–κ³  μžˆμ§€ μ•ŠκΈ° λ•Œλ¬Έμ—, 두 값이 λͺ¨λ‘ negative ν•œ 데이터인 t λŠ” λ¬΄μ‹œν•΄λ²„λ¦°λ‹€. 

 

 

β–Έ asymmetric binary similarity : Jaccard coefficient 

 

 

 

 

 

 

•  Distance on Numeric Attributes Minkowski Distance 

 

L-h norm

 

⇨ h = 1 : Manhanttan 

⇨ h = 2 : Euclidean 

⇨ h → ∞ : supremum 

 

 

 

 

 

•  Common Properties of a Distance 

 

⇨ Positive definities 

⇨ Symmetry 

⇨ Triangle Inequality 

⇨ Dissimilarity 

 

 

 

•  Dissimilarity between Ordinal attributes

 

 

 

 

•  Cosine similarity 

 

 

•  Mixed types 

 

 

 

→ 각 νƒ€μž…μ— λ§žλŠ” μ‹μœΌλ‘œ κ²Œμ‚° 후에 λ‹€μ‹œ 톡합해쀀닀. 

 

 

→ κ°€μ€‘μΉ˜ : weight importance for the f-th attribute

 

728x90