Quickly looking at the code it seems like there's only one log statement:

https://github.com/doyaaaaaken/kotlin-csv/blob/8108e5bcc24d55d813868ca3a238aed59a16b628/src/commonMain/kotlin/com/github/doyaaaaaken/kotlincsv/client/CsvFileReader.kt#L48

Do we really need to pull an entire library for logging?

https://github.com/doyaaaaaken/kotlin-csv/blob/ed7a6783313b3d033f3ffe702227b7a3961bc72b/build.gradle.kts#L50

I'm an Android user and currently that log would go basically nowhere.

Currently the only way to interact with a CSV is to parse all rows. Two use-cases that this does not cover are:

• Reading only the header. This is useful if you wish to provide a breakdown of what is included in the file. While it should be trivial to do without a library, the existence of this library and its parsing logic supports the position that this is a non-trivial task.
• Reading row-by-row, which is arguably a superset of the former use-case. This would be helpful when interacting with asynchronous workflows. One could attempt to read a single row from a piped input stream, and the library throws an exception when another line cannot be read in its entirety (as it does now with the full text). The producer can then continue to populate the input stream as data becomes available. The end result would be an asynchronous stream of rows (which I am not suggesting should be included in this library, but these changes would make this possible).

Describe the bug If rows are not equal-sized an exception is thrown: com.github.doyaaaaaken.kotlincsv.util.CSVFieldNumDifferentException: Fields num seems to be 4 on each row, but on 2th csv row, fields num is 3.

To Reproduce

        csvWriter().open(csvFile) {
            writeRow(listOf("a"))
            writeRow(listOf("a", "b"))
        }
        csvReader().open(csvFile) {
           readAllAsSequence()
        }

Expected behavior Missing cells are treated as nulls or empty strings.

Environment

• kotlin-csv version 0.11.0
• OS: Android 10

Screenshots N/A

Please allow writing to csv file without having to close it. I have a streaming scenario where I need to write each data I get to csv. Closing and reopening after each batch would be suboptimal.

Thanks David

Describe the problem

It quite not a bug, but I'm stuck on this exception:

Caused by: java.lang.NoClassDefFoundError: mu/KotlinLogging
	at com.github.doyaaaaaken.kotlincsv.client.CsvFileReader.<init>(CsvFileReader.kt:21)
	at com.github.doyaaaaaken.kotlincsv.client.CsvReader.open(CsvReader.kt:129)
	at com.github.doyaaaaaken.kotlincsv.client.CsvReader.readAll(CsvReader.kt:48)

It happens only on Ubuntu. MacOS is still fine.

Environment

• kotlin-csv version: 0.11.0
• java version: java8
• kotlin version: 1.4.10
• OS: Ubuntu 18.04.5

Any suggestion on this? Thank you!

I think to be able to write a variable number of arguments is nice also without creating a list before hand.

example:

csvWriter().open(testFileName) {
    writeRow("a", "b", "c")
    writeRow("d", "e", "f")
    writeRow(1,2,3)
    writeRow(date1, date2, date3)
}

Although Kotlin already supports a variable argument in instantiating a list with initial values, I think this one will be cleaner and less boiler plate on the user end.

Describe the bug Some csv editors after creating new fields add only touched fields for example some csv editors on mac don't add extra empty fields:

For example we have csv file: name,age,gender,hobby Steve,19,male,sport

I'm will open this file and add one more user, but with empty gender, and hobby, in out business logic it's allowed. name,age,gender,hobby Steve,19,male,sport Emma,20

editor will save latest record without extra empty fields, and parser with throw error something like Fields num seems to be 4 on each row, but on 3th csv row, fields num is 2.

according configuration i'm can ignore or throw error in such case, but it's valid situation and. I'm need save this record.

To Reproduce Steps to reproduce the behavior.

val record = mutableListOf<List>() csvReader { charset = "UTF-8" quoteChar = '"' delimiter = separator }.open(file) { readAllAsSequence().forEach { record.add(it) } }

1. export file: name,age,gender,hobby Steve,19,male,sport

2. add one more user, and try to read this file name,age,gender,hobby Steve,19,male,sport Emma,20

Actual behavior Fields num seems to be 4 on each row, but on 3th csv row, fields num is 2.

Expected behavior Reading file successful empty fields getting considered as empty one.

P.S. Will be good to add to class InsufficientFieldsRowBehaviour enum: IGNORE_MISSING_FIELDS and handle it, like it some fields are missing just consider them as empty one.

Environment

• kotlin-csv version [e.g. 0.10.0]
• java version [e.g. java8]
• OS: [e.g. MacOS]

Screenshots If applicable, add screenshots to help explain your problem.

Good morning,

I'm trying to use kotlin-csv on a comma delimited file (input stream) but I think there is a problem with managing the spaces and colon.

In particular, these are the first lines of the file:

Device serial,Date ,Temperature 51 (Medium) °C 11869,2021-02-09 00:14:59,7.2 11869,2021-02-09 00:30:01,7.1 11869,2021-02-09 00:44:59,7.2 11869,2021-02-09 00:59:59,7.4 11869,2021-02-09 01:14:58,7.5 11869,2021-02-09 01:29:58,7.5 11869,2021-02-09 01:44:58,7.3 11869,2021-02-09 01:59:58,7.2 11869,2021-02-09 02:14:58,7.2 11869,2021-02-09 02:29:58,7.2 11869,2021-02-09 02:44:58,7.2 11869,2021-02-09 02:59:57,7.3 11869,2021-02-09 03:14:57,7.2 11869,2021-02-09 03:29:57,7.3 11869,2021-02-09 03:44:57,7.2 11869,2021-02-09 03:59:57,7.4

..while this is the script:

data class DataClass( val FirstColumn: String, val SecondColumn: String, val ThirdColumn: String )

fun parse(data:InputStream): List?{

 val list = mutableListOf<DataClass>()

 try {
    val rows: List<List<String>> = csvReader().readAll(data)

    for (i in rows) {

 var firstColumn : String =  i[0]
 var secondColumn : String =  i[1]
 var thirdColumn : String =  i[2]

 list.add(DataClass(FirstColumn =firstColumn, SecondColumn = secondColumn,  ThirdColumn = thirdColumn))
    }

} catch (e: Exception) { e.printStackTrace() } return list }

Unfortunately only the first column of each row is correctly identified in the output, for example (first row): first column: 11869 second column: 2021-02-09 third colum: 0 No other colums detected.

Where is my mistake? Thank you

Current writeNext function have unnecessary map method. It cause unnecessary memory usage.

  private fun writeNext(row: List<Any?>) {
      val rowStr = row.map { field ->
          if (field == null) {
              ctx.nullCode
          } else {
              attachQuote(field.toString())
          }
      }.joinToString(ctx.delimiter.toString())
      writer.print(rowStr)
  }

Performance test

Result : Java Heap Space out error

Remove unnecessary map method.

  private fun writeNext(row: List<Any?>) {
      val rowStr = row.joinToString(ctx.delimiter.toString()) { field ->
          if (field == null) {
              ctx.nullCode
          } else {
              attachQuote(field.toString())
          }
      }
      writer.print(rowStr)
  }

Performance test

The test code was excluded for fear of slow execution for test. If you want to measure performance test, the performance test must be isolate alone. Because of, All tests execute independently and pararell then they use some memory. So, it might be on unexpected result

TEST CODE

"does not throw java heap space exception when execute writeRows" {
    val rows = mutableListOf<String>()

    for (i in 0..100000000) {
        rows.add(i.toString())
    }

    val sequence = listOf<List<String>>(rows).asSequence()

    assertDoesNotThrow {
        csvWriter().open(testFileName) {
            writeRows(sequence)
        }
    }
}

To reproduce

Have a csv file with header row with 3 columns and two data rows, first data row with two columns, second - with three. Like this:

First name | Last name | Citizenship
--------|----|---- John | Bobkins Michael| Pepkins| US

While invoking 'readAllWithHeaderAsSequence' on this file, the CSVFieldNumDifferentException is thrown saying that two colums are expected but three are found. It happens because 'fieldsNum' variable in the CsvFileReader.kt is initialized based on the first data row, while it has to be initialized based on the header row.

Expected behavior The following code has to return two rows:

csvReader().open(filePath) {
                readAllWithHeaderAsSequence().forEach {

. . . 
}

Environment

• kotlin-csv version 0.11.1
• java version - java8
• OS: Windows 10

...allocating new StringBuilder

This reuses the StringBuilder's internal buffer rather than reallocating for each field. On my local benchmark this improves performance of reading 48MB file by about 20%.

As discussed in #111 , to Introduce InsufficientFieldsRowBehaviour.EMPTY_STRING option on CSV reader seems good. This option treat field's value as empty String when the row has insufficient number of fields.

For example, If you read the below csv,

header1,header2,header3
a,b,c
x,y

you could get this.

listOf(
  listOf("header1", "header2", "header3"),
  listOf("a", "b", "c"),
  listOf("x", "y", ""),  // empty String value added on the end
)

I realize that generally it doesn't make sense to keep an input stream open when the end of the stream has been reached, but I'd like to read multiple CSV files that are inside a zip file (without extracting them first). However, that's currently not possible since the input stream gets closed when the end of the current zip entry is reached.

Problem

If we use readNext method, some options (e.g. excessFieldsRowBehaviour option) are bypassed and invalid.

There does not appear to be a use case for this method, so we are considering making it a private method. If you have feedback, please comment.

Hey there,

thank you very much for this gerat project.

Microsoft applications, for some reason, seem to require a BOM to parse for example UTF-8 files correctly, even though there is no byte order in UTF-8 like there is in 16/32. In order to open a created csv file correctly I suggest to add this special BOM (UTF-8 does require three special bytes 0xEF, 0xBB and 0xBF at the start of the file), even though the csvWriter is configured with the Charsets.UTF_8.name().

Why this is undocumented and why Excel seems to require a BOM for UTF-8 I don't know; might be good questions for Excel team at Microsoft.

What do you think or do you have any suggestion to solve this problem?

readAllWithHeader yields a List<Map<String,String>> and hence empty columns are being read as empty strings, so that we get "" for both col1 and col2 in the following example:

"col1","col2"
"",

I'd really like to get null for col2 here (this of course only makes sense if all strings are quoted, otherwise it wouldn't be clear how to interpret empty columns). I understand that you can't change the result to List<Map<String,String?>> now, but maybe you could add a nullCode option for reading as it already exists for writing. The default value is an empty string "" (=current behavior). I could then simply do

val nullCode = "NULL"
val rows = csvReader(nullCode=nullCode).readAllWithHeader(inputStream)
    .map { row -> row.mapValues { col -> if (col.value == nullCode) null else col.value } }

At first glance it seems that it only requires to change https://github.com/doyaaaaaken/kotlin-csv/blob/c23a51b98fbbcb1348b3928d561f1f5e11fba965/src/commonMain/kotlin/com/github/doyaaaaaken/kotlincsv/parser/ParseStateMachine.kt#L36-L48 to

                    delimiter -> {
                        field.append(nullCode)
                        flushField()
                        state = ParseState.DELIMITER
                    }
                    '\n', '\u2028', '\u2029', '\u0085' -> {
                        field.append(nullCode)
                        flushField()
                        state = ParseState.END
                    }
                    '\r' -> {
                        if (nextCh == '\n') pos += 1
                        field.append(nullCode)
                        flushField()
                        state = ParseState.END
                    }

and the same for https://github.com/doyaaaaaken/kotlin-csv/blob/c23a51b98fbbcb1348b3928d561f1f5e11fba965/src/commonMain/kotlin/com/github/doyaaaaaken/kotlincsv/parser/ParseStateMachine.kt#L87-L99

but I didn't check it thoroughly.