Purpose of this is to learn to create audit trail module.

Overview

Redis Trail

An effective audit trail solution can be crucial to an organization's security and data integrity as it can help find the who, what, and when. This helps organizations keep track of changes and investigate potential mistakes or violations of policy. Redis Trail is built as an audit trail application using Spring Boot framework for the backend and Redis as stream storage. And, the client application can integrate with Redis Trail using RESTful API to keep and monitor track of change.

1. Publish New Product Record to Stream Create A New Product Record

2. Publish New Edited Product Record to Stream Edit Product Record

3. Filter Product Record History By Timestamp Filter Product Record

4. Fetch All Product's Change History Fetch All Product Records

Overview video

Here's a short video that explains the project and how it uses Redis:

Embed your YouTube video

Technical Stack

  • Backend - Spring Boot, Redis
  • Programming Language - Kotlin

How it works

To illustrate how Redis Trail works, we have built a simple client application that makes use of the APIs provided by Redis Trail. The application allows the user to create or update a product record, when the product is created or updated, the information is sent to Redis Trail by calling API provided by Redis trail. The change record is stored in the Redis as a stream with a date time stamp, and it can be accessed by the user through API provided by Redis Trail to monitor the changes of a specific ID of the product during a certain timestamp as required.

Redis Trail creates a stream per product ID to persist the changes of each ID independently and thus makes it possible to identify exactly which user has done what changes on a specific product at a specified timestamp easily. Please refer to the Process Flow diagram for more details.

Architecture

Redis Trail Architecture

Internal Process Flow

Process Flow REST

Record Schema

Below is the schema of the Product's Record of the client application. Note: the subject must follow the below convention, where it tells what is the subject, ID of the subject, action, author, and the metadata of the change so that we can tell what exactly is happening to our subject. And, the subject in the demo refers to Product.

data class RecordEvent(
    var subject: String, // We will put "PRODUCT" as the example subject of record to audit
    val subjectId: Long, // Subject ID is the product's id in the client application that was called to Redis Trail to save.
    val action: String, // This can be defined by client application. We'd prefer "CREATE" or "UPDATE" as the action value.
    val data: Map<String, Any>, // The data of product
    val createdBy: Long, // The client application's user that makes change to the product record
    var createdAt: Long // The client application's timestamp that tell Redis Trail when that product data was made change.
)

Whenever a user makes any changes to the product data such as price, quantity, name, ..., etc. The client application calls POST API provided by Redis Trail to keep records of change:

Request Example

{
    "subject": "PRODUCT",
    "subjectId": 1,
    "action": "UPDATE",
    "data": {
        "name": "Teddy Bear",
        "qty": 100,
        "price": 22.00,
        "size": "small"
    },
    "createdBy": 1,
    "createdAt": 1661325276076
}

After receiving the request from the client application, Redis Trail converts the JSON request body to a stream in the Redis (key: RECORD_EVENT, group: RECORD_GROUP). And, Redis Trail itself is a subscriber of RECORD_EVENT and of the group RECORD_GROUP. And thus, once the stream is created, the Redis Trail receives the event, and creates a new stream with a key that combines between subject and its ID in Redis, making it possible to retrieve all logs by its "subject_id" (e.g. PRODUCT_1) in a later stage via GET API :

Response Example:

{
        "stream": "PRODUCT_1",
        "value": {
            "subject": "PRODUCT",
            "subjectId": 1,
            "action": "UPDATE",
            "data": {
                "name": "Teddy Bear",
                "price": 22.0,
                "qty": 100,
                "size": "small"
            },
            "createdBy": 1,
            "createdAt": 1661325276076,
            "publishTimestamp": "1661325276015"
        },
        "id": {
            "sequence": 0,
            "timestamp": 1661325276076,
            "value": "1661325276076-0"
        }
    }

Initialization

The demo data is prepared using two operations: Create and Update.

Create Product:

{
  "subject": "PRODUCT",
  "subjectId": 1,
  "action": "CREATE",
  "data": {
    "name": "Teddy Bear",
    "qty": 100,
    "price": 20.00,
    "size": "small"
  },
  "createdBy": 1,
  "createdAt": 1661325226321
}

Update Product:

{
    "subject": "PRODUCT",
    "subjectId": 1,
    "action": "UPDATE",
    "data": {
        "name": "Teddy Bear",
        "qty": 100,
        "price": 22.00,
        "size": "small"
    },
    "createdBy": 1,
    "createdAt": 1661325276076
}

Redis is mainly used as the streaming data for the record event store from the client application to Redis Trail.

How the data is stored:

All the record changes are stored in Redis as stream.

When Redis Trail application starts up successfully, it creates a stream with key RECORD_EVENT with group RECORD_GROUP. When the client application sends POST request to Redis Trail, the RedisTrail app creates a stream in the Redis: XADD {stream_key} {timestamp}-0 {record_event_data}

  • For Example: XADD RECORD_EVENT 1661400135369-0 {RECORD_EVENT_DATA}

Redis Trail itself is a subscriber of RECORD_EVENT and of the group RECORD_GROUP. And thus, once the above stream is created, the Redis Trail receives the event, and creates a new stream with a key that combines between subject and its ID in Redis, making it possible to retrieve all logs by its "subject_id" (e.g. PRODUCT_1): XADD {subject}_{subjectId} MAXLEN 100 {timestamp}-0 {data_of_subject}

  • For Example: XADD PROUDCT_1 MAXLEN 100 1661400135369-0 {data_of_subject}

Notice: MAXLEN 100 indicates that we can store the latest 100 change history records of one id per subject (e.g. PRODUCT_1), and the old data of that subject will be removed once the change history record reaches MAXLEN. The MAXLEN configuration can be configured dynamically in future enhancement, and the old data could be stored in a NOSQL database as a backup. This is an effective way to reduce the size and saves the cost of Redis storage.

How the data is accessed:

The client application can send REST API requests to Redis Trail in order to retrieve the change history records of a specific product in a specific timestamp, and Redis Trail fetches the data from the Redis and responses back to the client:

  • To fetch all record event change history XRANGE {stream_key} - +
    • For Example: XRANGE PRODUCT_1 - +
  • To filter for a specific timestamp range XRANGE {sream_key} {from_timestamp} {to_timestamp}
    • For Example: XRANGE PRODUCT_1 1661400135369 1661400135569

How to run it locally?

Prerequisites

  • Postman - v9.*
  • JAVA - v11
  • Docker - v19.03.13

Local installation

Clone Redis Trail repository and go to ./redis-trails folder(cd ./redis-trails) and then:

# Run Redis Stack in Local
docker-compose up -d

# Run Redis Trail
./gradlew bootRun

Test Redis-Trail

Open Postman and Import Redis Trail Collection.json.

  1. Create Subject Change Record: POST - http://localhost:8080/api/v1/publish/stream
  2. Fetch Subject Change Record: GET - http://localhost:8080/api/v1/records/{subject}/{subject_id}

Future of Redis Trail

For future enhancement of Redis Trail, we are planning to make the backend a Java library, so that it can be easily integrated with the client application.

We are also planning to build another communication layer for the client application to talk to Redis in the Redis Trail via Redis Pub/Sub directly.

More Information about Redis Stack

Here some resources to help you quickly get started using Redis Stack. If you still have questions, feel free to ask them in the Redis Discord or on Twitter.

Getting Started

  1. Sign up for a free Redis Cloud account using this link and use the Redis Stack database in the cloud.
  2. Based on the language/framework you want to use, you will find the following client libraries:

The above videos and guides should be enough to get you started in your desired language/framework. From there you can expand and develop your app. Use the resources below to help guide you further:

  1. Developer Hub - The main developer page for Redis, where you can find information on building using Redis with sample projects, guides, and tutorials.
  2. Redis Stack getting started page - Lists all the Redis Stack features. From there you can find relevant docs and tutorials for all the capabilities of Redis Stack.
  3. Redis Rediscover - Provides use-cases for Redis as well as real-world examples and educational material
  4. RedisInsight - Desktop GUI tool - Use this to connect to Redis to visually see the data. It also has a CLI inside it that lets you send Redis CLI commands. It also has a profiler so you can see commands that are run on your Redis instance in real-time
  5. Youtube Videos
You might also like...
Eton - Note taking app made to learn some good practices

Eton 📜 Description Note taking app made to learn some good practices, from Phil

A sample Music Player project that help you learn about Compose in Android
A sample Music Player project that help you learn about Compose in Android

Music App Compose UI A sample Music Player project that help you learn about Compose in Android. Note that this app only contain UI and has no logic.

Showify is a my first simple ✅ Android application 📱 using DI, where I learn how to use dagger-hilt, retrofit2, mvvm, livedata, Requestly Interceptor
Showify is a my first simple ✅ Android application 📱 using DI, where I learn how to use dagger-hilt, retrofit2, mvvm, livedata, Requestly Interceptor

Showify is a my first simple ✅ Android application 📱 using DI, where I learn how to use dagger-hilt, retrofit2, mvvm, livedata, Requestly Interceptor and so much more...

Learn Kotlin, easy bites at a time
Learn Kotlin, easy bites at a time

Welcome! I hope you're having an amazing day! 🚀 This repository is a reference of how I think one should approach learning kotlin step-by-step. Insid

Tools & tips to learn about recomposition in Jetpack Compose

⚡ Recomposition in Jetpack Compose List of practical tips and code snippets to avoid unnecessary recomposition in Jetpack Compose. This is an active r

For Kotlin with SpringBoot project that have multi-module-structure template

Goals kotlin + spring-boot + gradle + multi-module building Module-Structure ---root |--- src.main.kotlin.KotlinSpringbootMultiModuleTemplateAppl

FragmentContainerViewIdBugDemo - minimal repro project demonstrating a bug in FragmentContainerView's id check logic in the context of a dynamic feature module

FragmentContainerViewIdBugDemo minimal reproduce project demonstrating an apparent bug in FragmentContainerView's id check logic in the context of a d

A complete Kotlin application built to demonstrate the use of Modern development tools with best practices implementation using multi-module architecture developed using SOLID principles
A complete Kotlin application built to demonstrate the use of Modern development tools with best practices implementation using multi-module architecture developed using SOLID principles

This repository serves as template and demo for building android applications for scale. It is suited for large teams where individuals can work independently on feature wise and layer wise reducing the dependency on each other.

Clean Android multi-module offline-first scalable app in 2022. Including Jetpack Compose, MVI, Kotlin coroutines/Flow, Kotlin serialization, Hilt and Room.

Android Kotlin starter project - 2022 edition Android starter project, described precisely in this article. Purpose To show good practices using Kotli

Owner
Soramitsu Khmer Labs
OSS / Experimental / R&D for Soramitsu Khmer
Soramitsu Khmer Labs
Portfolio application for the purpose of listing events according to the return of an API

My Event Schedule Aplicativo portifólio com a finalidade de listar eventos de ac

Ricardo Souza 2 Feb 26, 2022
An Android UI clone of ABA Mobile app for education purpose only.

ABAUIClone An Android UI clone of ABA Mobile app for education purpose only. Project Specs Language: Kotlin UI Design system: XML-based Design pattern

UTNGY Pisal 2 Sep 6, 2022
A toy port scanner to help me (and you!) learn Kotlin + Akka.

kotlin-akka-portscan A toy program to help me (and you!) learn Kotlin + Akka. butwhy.gif When I want to learn a new language, I've found it helpful to

Jeremi M Gosney 4 Jul 23, 2022
Learn how to make an app designed for single-screen devices shine when running on foldable and dual-screen devices

dcberlin21-workshop Make your app shine om foldable devices with the samples we have here. Related links SDK open-source code SDK samples (Kotlin) App

Cesar Valiente 3 Oct 26, 2021
A basic, incomplete, buggy, far from efficient UI toolkit for Kotlin/Android. An experiment for fun and to learn.

Apex Apex is just a simple proof of concept to demonstrate how easily you can build your own UI Toolkit from scratch. This code base is most likely fu

Romain Guy 79 Sep 7, 2022
A basic, incomplete, buggy, far from efficient UI toolkit for Kotlin/Android. An experiment for fun and to learn.

Apex Apex is just a simple proof of concept to demonstrate how easily you can build your own UI Toolkit from scratch. This code base is most likely fu

Romain Guy 79 Sep 7, 2022
A project to learn about Reactive Microservices experimenting with architectures and patterns

reactive-microservices-workshop Copyright © 2021 Aleix Morgadas - Licenced under CC BY-SA 4.0 A project to learn about Reactive Microservices experime

Aleix Morgadas 7 Feb 21, 2022
A single screen app learn in google basic Android Development course.

Project: Lemonade App - Starter Code Starter code for the first independent project for Android Basics in Kotlin Introduction This is the starter code

Kaushal Raj 1 Dec 9, 2021
PlanetFacts - An educational android app for kids to learn about the planets in our solar system. Built with Kotlin.

PlanetFacts PlanetFacts is an offline simple, modern & material-designed educational Android application for kids. It contains basic facts with visual

Saikat Datta 1 May 16, 2022
Learn-kotlin - Learning more about Kotlin in various content

Kotlin study roadmap https://kotlinlang.org/docs/reference/ Getting Started Basi

Danilo Silva 0 Jan 7, 2022