Custom partitioner for Spring Batch

Overview

Custom partitioner for Spring Batch

Who is it for?

For cases in which you have multiple files to process as part of the batch, Spring Batch offers the ready to use MultiResourcePartitioner, which sets up one ExecutionContext for each Resource, making it possible to process multiple files in parallel.

Some use cases could go even further, and also partition each single file, but there is no built-in partitioner that is designed to do anything like that. In this library you will find an implementation of the Partitioner and an extension of the FlatFileItemReader to do just that, giving you the possibility to improve the performance of your batch processing even further, if your specific use case allows it.

⚠️ : Since the order of execution of the partitions is not guaranteed, use this library only if the order in which the lines of the file are processed doesn't matter

How to use it

Using the MultiResourceChunkedPartitioner is pretty straight forward, and very similar to how you would use the standard MultiResourcePartitioner.

The configuration of the partitioner should look something like this:

Kotlin

fun partitioner(resources: List<Resource>): MultiResourceChunkedPartitioner {
	val partitioner = MultiResourceChunkedPartitioner(resources)

	partitioner.setLinesToSkip(1)   // Set in the partitioner instead of the ItemReader
	partitioner.partitionSize = 10_000  // Sets the number of lines to process in each partition
	return partitioner
}

Java

public class PartitioningStep   {
    
    MultiResourceChunkedPartitioner partitioner(ArrayList<Resource> resources) {
        MultiResourceChunkedPartitioner partitioner = new MultiResourceChunkedPartitioner(resources);
    
        partitioner.setLinesToSkip(1);   // Set in the partitioner instead of the ItemReader
        partitioner.setPartitionSize(10_000);  // Sets the number of lines to process in each partition
    
        return partitioner;
    }
}

The MultiResourceChunkedPartitioner adds three key-value pairs to each ExecutionContext:

  • fileName - Same as the MultiResourcePartitioner
  • startingLineIndex - The index of the line from which the ItemReader that will take that partition should start reading from
  • endingLineIndex - The index of the line from which the ItemReader that will take that partition should stop reading at

If the partitionSize is not set, then the MultiResourceChunkedPartitioner will create one partition per file, behaving in the same way as the MultiResourcePartitioner.

The PartitionedFlatFileReader is designed to integrate easily with the MultiResourceChunkedPartitioner. The configuration of the reader should look something like this:

Kotlin

{ val reader = PartitionedFlatFileReader() reader.setResource(FileSystemResource(pathToFile.substringAfter("file:/"))) reader.setLinesToRead(startingLineIndex, endingLineIndex) reader.setLineMapper { it, idx -> // Line mapping } return reader }">
@Bean
@StepScope
fun <T> reader(
    @Value("#{stepExecutionContext[fileName]}") pathToFile: String,
    @Value("#{stepExecutionContext[startingLineIndex]}") startingLineIndex: Int,
    @Value("#{stepExecutionContext[endingLineIndex]}") endingLineIndex: Int,
): PartitionedFlatFileReader<T> {
    val reader = PartitionedFlatFileReader<T>()
    
    reader.setResource(FileSystemResource(pathToFile.substringAfter("file:/")))
    reader.setLinesToRead(startingLineIndex, endingLineIndex)
    reader.setLineMapper { it, idx ->
        // Line mapping
    }
    
    return reader
}

Java

reader = new PartitionedFlatFileReader(); reader.setResource(new FileSystemResource(pathToFile.substring(pathToFile.lastIndexOf("file:/") + 1))); reader.setLinesToRead(startingLineIndex, endingLineIndex); reader.setLineMapper( (row, idx) -> { // Line mapping } ); return reader; } }">
public class PartitioningStep {

    @Bean
    @StepScope
    PartitionedFlatFileReader<T> itemReader(
            @Value("#{stepExecutionContext[fileName]}") String pathToFile,
            @Value("#{stepExecutionContext[startingLineIndex]}") int startingLineIndex,
            @Value("#{stepExecutionContext[endingLineIndex]}") int endingLineIndex
    ) {
        PartitionedFlatFileReader<T> reader = new PartitionedFlatFileReader<T>();

        reader.setResource(new FileSystemResource(pathToFile.substring(pathToFile.lastIndexOf("file:/") + 1)));
        reader.setLinesToRead(startingLineIndex, endingLineIndex);
        reader.setLineMapper(
                (row, idx) -> {
                    // Line mapping
                }
        );

        return reader;
    }
}

The PartitionedFlatFileReader behaves very much in the same way as the FlatFileItemReader, with the noticeable differences being the method setLinesToRead which should take as parameters the values that the partitioner added to the ExecutionContext, and the fact that the method setLinesToSkip is deprecated, since the lines to skip should be set at the partitioner level, to avoid skipping the lines for all the partitions of the same file, and not just for the first partition.

Notes

In case of conflicts, the default key names for filekeyName, startingLinekeyName and endingLinekeyName can be overridden.

You might also like...
A custom Stepper for jetpack compose πŸš€
A custom Stepper for jetpack compose πŸš€

Compose-Stepper Compose-Stepper library provides a custom stepper in the modern android toolkit Jetpack compose which can be easily added in to your c

Execute asynchronous batch tasks with predefined or custom UI in Android.
Execute asynchronous batch tasks with predefined or custom UI in Android.

AndroidBatchWorker Execute asynchronous batch tasks with predefined or custom UI in Android. Import Add JitPack repository to your project level build

Spring-graphql-getting-started - Spring for GraphQL provides support for Spring applications built on GraphQL Java

Getting Started with GraphQL and Spring Boot Spring for GraphQL provides support

This repository contains RabbitMQ Protobuf starters with its usage samples for spring-rabbit and spring-cloud-starter-stream-rabbit modules

This repository contains RabbitMQ Protobuf starters with its usage samples for spring-rabbit and spring-cloud-starter-stream-rabbit modules

Spring-with-maven - Spring Boot App with Postgresql and maven

Spring Boot Api Aplikasi ini dibuat menggunakan bahasa kotlin dan untuk database

Reactive setup with Spring WebFlux , Kotlin, Postgres and Spring Data R2DBC

Reactive Spring with Kotlin and Pg Spring WebFlux with Netty instead of Spring Web with Tomcat Mono and Flux in all layers (controller, service, repo)

Spring-kotlin - Learning API Rest with Kotlin, Spring and PostgreSQL

Kotlin, Spring, PostgreSQL and Liquibase Database Migrations Learning Kotlin for

Android login spring - Android login against spring backend

Android Jetpack Compose login implementation with JWT tokens against our own bac

Implementation of ExpandableListview with custom header and custom content.
Implementation of ExpandableListview with custom header and custom content.

ExpandableLayout ExpandableLayout provides an easy way to create a view called header with an expandable view. Both view are external layout to allow

Android layout decorators : Injecting custom attributes in layout files, Using decorators to get rid of unnecessary class explosion with custom views
Android layout decorators : Injecting custom attributes in layout files, Using decorators to get rid of unnecessary class explosion with custom views

Decor Decor is a library that applies decorators to Android layout with additional attributes without the need to extend and create a custom View for

Custom View classes for TextView, EditText & Buttons - to set custom fonts
Custom View classes for TextView, EditText & Buttons - to set custom fonts

CustomFontView Custom font classes for TextView, EditText & Buttons How to integrate the library in your app? Gradle Dependecy dependencies {

StartPointSeekBar is a custom view for the Android platform that makes it possible to have a SeekBar to have custom start point.
StartPointSeekBar is a custom view for the Android platform that makes it possible to have a SeekBar to have custom start point.

Forked/Inspired from https://code.google.com/p/range-seek-bar/ by [email protected] This solves the problem as described in http://

CuteToast is an Material Design Custom Toast for Android | Custom Material Design Toast
CuteToast is an Material Design Custom Toast for Android | Custom Material Design Toast

CuteToast is an Android Custom Toast library that could be used instead of Default Toast. It does everything as Toast but with some extra spice.

PagedGrid - Custom android view composed by multiple page grids with custom content and layout
PagedGrid - Custom android view composed by multiple page grids with custom content and layout

PagedGrid A PagedGrid is a ViewPager which pages are GridLayout with equal distributed rows and columns. This project is an Android library, written i

Custom-view-animated-file-downloader - Custom Views, Animations, Broadcast Receivers, Notifications
Custom-view-animated-file-downloader - Custom Views, Animations, Broadcast Receivers, Notifications

Downloader App Custom views , Drawing with Canvas, Animations (with motionlayout

πŸš€πŸ§¨πŸ“ Series of Tutorials to learn about Jetpack Compose with subjects Material Widgets, Layout, SubcomposeLayout, custom layouts, State, custom rememberable, recomposition, LaunchedEffect, side-effects, Gesture, Animation,  Navigation, Canvas, UIs like whatsapp and others.
A Java library that models spring dynamics and adds real world physics to your app.

Rebound About Rebound is a java library that models spring dynamics. Rebound spring models can be used to create animations that feel natural by intro

Support for Spring's RestTemplate within native Android applications

Spring for Android Spring for Android is a library that is designed to provide components of the Spring Framework family of projects for use in native

A spring indicator like Morning Routine guide.
A spring indicator like Morning Routine guide.

SpringIndicator An indicator like Morning Routine guide.It was originally based on BezierDemo. The sample app: click me #Usage Add the dependency to y

Releases(v1.0.0-beta1)
Owner
Lorenzo Milicia
Lorenzo Milicia
Successor to ProxyBuilder - Uses Spring & Netty for testing proxies, interacting with a local MariaDB.

Successor to ProxyBuilder - Uses Spring & Netty for testing proxies, interacting with a local MariaDB.

Kai o((>Ο‰< ))o 8 Dec 6, 2022
Spring Kotlin Design Patterns

Spring Kotlin Design Patterns This is a project for learning design patterns in Kotlin Sources 1- GURU Requirements Java 11+ IntelliJ IDEA / Netbeans

Gustavo 2 Nov 22, 2022
Spring boot web + Kotlin template project

kotpringboot-multimodule-template Table of Contents Overview How to run Overview How to run Run with profiles: Note that default spring profile would

Hwan Jo 7 Oct 30, 2022
Embeddable custom voice assistant for Android applications

Aimybox voice assistant Open source voice assistant built on top of Aimybox SDK iOS version is available here Key Features Provides ready to use UI co

Just AI 176 Jan 2, 2023
Carousel Recyclerview let's you create carousel layout with the power of recyclerview by creating custom layout manager.

Carousel Recyclerview Create carousel effect in recyclerview with the CarouselRecyclerview in a simple way. Including in your project Gradle Add below

Jack and phantom 514 Jan 8, 2023
Custom ViewPager that allows to block left or right swipe gestures.

SwipeDirectionViewPager Introduction Custom ViewPager that allows to block swiping right or left where the ViewPager child fragments set the scroll di

Jan Rabe 10 Nov 9, 2021
Some fancy custom views for kotlin

CoolCustomViews 1. Neruromorphic ProgressBar Resources Documentaion and Other So

Siddharth sharma 16 Dec 16, 2022
This is a Kotlin multiplatform template project used to generate and deploy a natively compiled AWS lambda function using the custom runtime.

Overview This is a Kotlin multiplatform template project used to generate and deploy a natively compiled AWS Lambda function using a custom runtime. U

Greg Steckman 5 Jun 25, 2022
This sample Kotlin app shows a list of custom shoes added by the users

Shoe-store This sample Kotlin app shows a list of custom shoes added by the users. The app displays the content with RecyclerView and uses a tradition

Ana Stanescu 2 Aug 27, 2022
Custom Sneaker view for Android.

SneakerView How to install ? You can add the library to your project using jitpack.io. Add the code below to your project's settings.gradle file. all

Osman GΓΌl 3 Aug 26, 2022