BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News Processing Data with Kotlin Dataframe Preview

Processing Data with Kotlin Dataframe Preview

Kotlin DataFrame, available as a first public preview, is a new library for processing data from any source, such as CSV, JSON, Excel and Apache Arrow files. The DataFrame library works together with Kotlin data classes and hierarchical data schemas by using a Domain Specific Language (DSL).

A data frame is a two dimensional table with labeled columns, comparable to spreadsheets, SQL tables, or CSV files. The Kotlin DataFrame GitHub repository contains more information about the set of operations on the data, available through a DSL, supporting data analysis. The library started as a wrapper around the Krangl library, but most of the functionality was rewritten over time.

The Kotlin DataFrame library was designed to read and display data from any source and allows nesting columns and cells. Any Kotlin object or collection can be stored and retrieved from a DataFrame.

Data may be supplied via a file such as students.csv:

firstName, lastName, country
Akmad, Desiree, The Netherlands
Serik, Chuy, India
Ioel, Jan, Belgium
Draco, Arti, Argentina
Myrna, Hyginos, Bolivia
Dalila, Dardanos, Belgium

The DataFrame is created based on the contents of the file:

val studentsDataFrame = DataFrame.read("students.csv")
print(studentsDataFrame.head())

By default, the head() method returns the first five rows:

    firstName lastName         country
 0 	    Akmad  Desiree The Netherlands
 1 	    Serik     Chuy           India
 2       Ioel      Jan         Belgium
 3 	    Draco 	  Arti       Argentina
 4 	    Myrna  Hyginos         Bolivia

Alternatively, the DataFrame may be created programmatically:

val firstName by columnOf("Akmad", "Serik", "Ioel", "Draco", "Myrna", "Dalila")
val lastName by columnOf("Desiree", "Chuy", "Jan", "Arti", "Hyginos", "Dardanos")
val country by columnOf("The Netherlands", "India", "Belgium", "Argentina", "Bolivia", "Belgium")

By supplying the head() method with an argument, the number of elements may be defined, in this case two:

val customDataFrame = dataFrameOf(firstName, lastName, country)
print(customDataFrame.head(2))
    firstName  lastName          country
 0 	    Akmad   Desiree  The Netherlands
 1 	    Serik      Chuy            India

The API offers various ways to retrieve specific types of data such as the contents of the first row:

println(studentsDataFrame.get(0).values())
[Akmad, Desiree, The Netherlands]

Alternatively, a specific column, such as the country column may be retrieved:

println(studentsDataFrame.getColumnOrNull(2)?.values())
[The Netherlands, India, Belgium, Argentina, Bolivia, Belgium]

More advanced API methods allow, for example, sorting and removing elements from a DataFrame:

println(studentsDataFrame.sortBy("firstName").remove("country"));
    firstName  lastName
 0 	    Akmad   Desiree
 1	   Dalila  Dardanos
 2 	    Draco 	   Arti
 3       Ioel       Jan
 4 	    Myrna   Hyginos
 5 	    Serik 	   Chuy

A DataSchema annotation may be specified to improve the data handling:

@DataSchema
interface Student {
   val firstName: String
   val lastName: String
   val country: String
}

Now the schema can be used together with the DataFrame API to filter on the country field of Student and sort on the firstName field of Student:

val studentsDataFrame = DataFrame.read("students.csv")

println(studentsDataFrame.filter{it[Student::country] != "Belgium"}.sortBy(Student::firstName))
    firstName  lastName         country
 0 	    Akmad   Desiree The Netherlands
 1 	    Draco 	   Arti       Argentina
 2 	    Myrna   Hyginos         Bolivia
 3 	    Serik      Chuy           India

More information can be found in the first video about Kotlin DataFrame which covers the basic operations and processing tables. Various examples are available and the #datascience channel on Slack may be used to ask questions after signing-up.

About the Author

Rate this Article

Adoption
Style

BT