Buffl
Buffl
Data Curation

1 Week

LI
von Luca I.

Panda Structures

Series:

  • one dimensional array-like object

  • containing sequence of value and associated array of data labels -> index



DataFrame:

  • rectangular table of data

  • contains ordered collection of columns

  • each column can have a different value type

  • rows and clumns have index



Operation on series:

  • change/ assign index

  • get specific value/ assign value per index

  • filtering

  • scalar multiplication

  • math functions







Operation on series:

  • create series form dict





Operation on series:

  • create new order for sdata


-> as aveiro index is not present in sdata is is assigned th NaN value

-> as Alentejo is not present in the new index order it is not in the new series



+



=


Operation on series:

  • check for not null/ null values




Operation on series:

  • rename entire series and index




Operation on series:

  • get the whole series

  • get index

  • get the values


Whole Series:


Get Index:


Get Values:



Dataframe

dict -> dataframe

what happens if you pass a column that is not contained in the dict?

-> missing columns have NaN values

Selecting rows and columsn in DataFrame

  • either by dict-like notation or by attribute


DataFrame

  • retrieve by position or row (2002, Aveiro, Nan,Nan)

  • retrieve specific value (Aveiro)



Rows can also be retrieved by position or name with the special loc or iloc



First Index, than column

Assigning columns with values

  • each entry 16.5 / values 0 - 5

  • by index with specific values


In this case the debt column




values attribute with DataFrame

-> returns the data contained in the DataFrame as

a two-dimensional ndarray


Index and Column objects for Dataframes

  • Every Series and DataFrame has an Index object that holds the row labels (and a little metadata like a name).

  • df.columns is also an Index object—just for column labels.

  • Default row index is RangeIndex(0, 1, 2, …), but you can use anything hashable (strings, dates, tuples).

  • Index objects are immutable and thus can’t be modified by the user


Instead:




Reindexing:

Dataframe




-> creates new object with data conformed to new index



Reindexing coumns:


->


the columns keyword is used


dropping entries

DataFrame

  1. drop Porto and coimbra row

  2. drop two and two+four column


-> .drop() returns new object




-> to drop columns either use axis = 1 or axis = ‘columns’

using .drop() without creating new object

using in-place


Indexing, Selection and Filtering

  1. get Aveiro and Lisboa row

  2. get “two”, column

  3. get “three” and “One” column in this order






• Slicing or selecting data with a boolean array:

  1. get boolean array for all values in column three that are larger than 5

  2. get inly Lisboa, Coimvra and Porto








Since the 'c' and 'e' columns are not found in both DataFrame objects, they appear as all missing in the result. The same holds for the rows whose labels are not common to both objects.

Operations between DataFrame and Series



  • By default, arithmetic between DataFrame and Series matches the index of the Series on the DataFrame’s columns, broadcasting down the rows.

    -> would not work here as series indexes don’t match column index

index = match the series index based on row index

column (default), match series with column index

Function application and mapping

  • get the difference between max and min value for each column

  • convert each value in a row to a 2f


lambda is column wise, as different columns can have different values types (String, bollean, ec.) so it makes sense to automatically do it per column


  • apply: a function on one-dimensional arrays to each

    column or row

  • default, one column across rows

  • columns, one row across columns

  • applymap to perform element-wise operation on each value


Sorting

DataFrame

  1. by index

  2. by column

  3. column/ descending




Sorting Values

DataFrame


  • sorting values in column b

  • sorting values first based on b than on a




  • wie in Excel when man in Tabelle auf the Pfeil drückt


summary statistics for dataFrames


  • sum up values in each column

  • sum up value in rows

  • multiple summary statistics with one function







Available summary statistics


correlations and covariance

DataFrame




Unique values, value counts

  1. unique values

  2. count values and sort them

  3. count values without sorting




-> array(['c', 'a', 'd', 'b'], dtype=object) (not sorted)

-> .value_counts() automatically sorts


Membership

DataFrames


  1. get Mask array

  2. use mask to filter dataset



1.



  1. How is missing data in pandas represented?

  2. check for missing data, respective values


  • the floating-point value NaN (Not a Number)

  • df.isnull()



Na handling methods

  • .dropna() -> drops rows

    • .dropna(axis=1) -> drops columns

  • .fillna() -> fill missing data

  • .isnull() -> Returns a Boolean mask (True for missing values).

  • .notnull(), opposite of isnull -> returns

-> All create new object

-> Can modify in place: only dropna, fillna w. inplace=True

-> Never in-place: isnull, notnull.




Filtering out missing data per row:

  1. drop only rows with at least one missing value

  2. drop only rows wer all values are NaN



  1. df.data.dropna()

  1. df.data.dropna()


Filtering out null coluns

  1. remove this column




fillna() function

  1. replace NaN values with 0

  2. replace NaN values differently across columns

  3. fill NaN value with the value before







fillna arguments



Removing duplicates

  1. booean series indicating which rows are duplicates

  2. remove duplicates

  3. remove duplicates in specific column

  4. keeoming the last duplicate





Transforming Data using Function or Mapping

  1. add new column animal, that maps the correct animal to the beef type

  2. map the correct animal to the beef directly in the food column






Replacing Values

  1. one values with another value

  2. multiple values with one value

  3. multiple values with multiple values



1

2

3


Renaming Axis Indexes

1. rename lisboa with cascais and three with threeandahalf

  1. reanem porto with Braga and modify original dataset



Rename returns new DataFrame unless inplace = True



Discretization and binning

  1. bin the age in intervals 18,25,35,60,100

  2. get the interval number for each age number

  3. get interval indexes

  4. Assign 'Youth, YoungAdult, MiddleAged and Senior as bin names



1

2

3

4


Discretization and Binning

  1. equal length bins based on min and max values in the data

  2. based on quantiles





detecting and filtering outliers

  1. filter out all values larger than 3 for a specific columns

  2. filter out all values larger than 3 for the whole dataset






Computing indicator/ Dummy Variables:





Pandas String Manipulation

  1. array indicating which email uses gmail domain

  2. display the 0 to 4th element

  3. Uppercase all mail adresses

  4. replace gmail with yahoo

  5. get length of string




Pandas String Manipulation

  1. checking if characters in strings are just letters





Beitreten
Vorschau

Author

Luca I.

Informationen

Zuletzt geändert

Kurs melden
© 2023 Buffl GmbH