This application is used to adjust the scaling of kubernetes deployments depending on the message-count of given RabbitMQ queues.

Last update: May 9, 2022

PodAutoScaler

This application is used to adjust the scaling of kubernetes deployments depending on the message-count of given RabbitMQ queues.

It uses a YAML configuration where you can set all queue--deployment relations with the scaling-rules.

Deployment with Helm

To deploy the application using Helm, you simply write a values.yaml and add the mayope helm-repository. Then you can install it with the second command.

helm repo add mayope https://charts.mayope.net

helm upgrade --install <name of deployment> mayope/podautoscaler -f values.yaml -n <namespace to deploy to>

Content of the values.yaml:

image:
  repository: mfhelm/podautoscaler
  pullPolicy: Always
  version: 1.1.0

rabbitmqAccessSecret: <name of the secret for RabbitMQ (*1)>
rabbitmqUser: <username for RabbitMQ-Server>
rabbitmqHost: <host of RabbitMQ-Server>
rabbitmqPort: <port of RabbitMQ-Server>

logTrace: false

scalerConfig: |
  <scalerConfig>

*1: The secret must set the environment-variable rabbitmq-password to the password for the RabbitMQ-User.

The value scalerConfig must contain the string of the config-yaml. It can be written directly in the values.yaml or read from a file by appending --set-file scalerConfig=<path to config.yaml> to the helm-upgrade-command (in this case you must remove the last two lines).

Optionally tracing of rule-execution can be enabled by setting logTrace: true.

Configuration

The configuration has the following format:

-
  label: "<name for this conf entry (e.g. for logs)>; optional (default is '_unnamed_')"
  deploymentNamespace: "<kubernetes namespace>; optional (*2)"
  deployment: "<name of the deployment to scale>"
  interval: "<interval in which the queue should be checked>; number | string>"
  queues:
    -
      virtualHost: "<path to the virtual host for the queue (including leading slash)>; optional (default is '/')"
      name: "<name of the RabbitMQ queue to observe>"
      ruleset:
          type: "<limit | linearScale | logScale; sets the type of the rules>"
          rules:
            - # type = limit
              minMessageCount: "<minimum count of messages to trigger this limit>; number | string"
              podCount: "<number of pods to run when this limit is triggered>; number"
            - # type = linearScale
              factor: "<podCount = factor * messageCount>; number"
              stepThreshold: "<min diff of podCount to trigger update>; number; optional (default is 1)"
              minPodCount: "<lower limit for podCount>; number; optional (default: 1)"
              maxPodCount: "<upper limit for podCount>; number; optional (default: 10)"
            - # type = logScale
              base: "<podCount = log(messageCount, base) + offset>; number"
              offset: "<number; optional (default is 0)>"
              stepThreshold: "<min diff of podCount to trigger update>; number; optional (default is 1)"
              minPodCount: "<lower limit for podCount>; number; optional (default: 1)"
              maxPodCount: "<upper limit for podCount>; number; optional (default: 10)"
  • *2: default is the value of the java-property 'namespace' or (if not set) the value of the environment-var 'NAMESPACE'
  • value of interval: can be a number (in seconds) or a string of the following format <value>s|m|h|d where s means seconds, m minutes, h hours and d days
  • limit ruleset:
    • must have at least one rule
    • if there is nor rule with minMessageCount = 0 then one will be created with podCount of the rule with the smallest minMessageCount
    • value of minMessageCount: can be a number or a string of the following format <value>k|m where k means thousand an m million
  • linearScale ruleset:
    • must have exactly one rule
  • logScale ruleset:
    • must have exactly one rule
  • types of rules can not be mixed

The configuration is passed to the application via the environment-variable config (must be a string with the YAML content).

Example with limit

-
  label: "example_with_limit"
  deploymentNamespace: ns-1
  deployment: example
  interval: 60
  queues:
    -
      virtualHost: /hostA
      name: queue_a
      ruleset:
          type: limit
          rules:
            -
              minMessageCount: 0
              podCount: 1
            -
              minMessageCount: 500
              podCount: 2
            -
              minMessageCount: 2k
              podCount: 5

This example will observe the queue 'queue_a' on virtual-host '/hostA' and scale the deployment 'example' in namespace 'ns-1' according to the following rules:

Message-Count new Pod-Count
0 <= mc < 500 1
500 <= mc < 2000 2
mc >= 2000 5

Example with linearScale

-
  label: "example_with_linearScale"
  deploymentNamespace: ns-1
  deployment: example
  interval: 5m
  queues:
    -
      virtualHost: /hostA
      name: queue_a
      ruleset:
          type: linearScale
          rules:
            -
              factor: 0.01
              stepThreshold: 2
              minPodCount: 1
              maxPodCount: 5

This example is similar to the last. The difference is the interval (which is 5 minutes) and the rules. For the rules here are some example-calculations:

Message-Count current Pod-Count new Pod-Count Notes
mc = 490 pc = 1 5 pod-count will be rounded
mc = 700 pc = 1 5 because maxPodCount is 5
mc = 100 pc = 2 2 because stepThreshold is 2 and the new pod-count would be 1 (pc - 1 < stepThreshold)

Example with logScale

-
  label: "example_with_logScale"
  deploymentNamespace: ns-1
  deployment: example
  interval: 5m
  queues:
    -
      virtualHost: /hostA
      name: queue_a
      ruleset:
          type: logScale
          rules:
            -
              base: 10
              stepThreshold: 2
              minPodCount: 1
              maxPodCount: 5
Message-Count current Pod-Count new Pod-Count Notes
mc = 2 pc = 5 1 new Pod-Count will be rounded and minPodCount is 1
mc = 1000 pc = 1 3
mc = 30000 pc = 5 5 because stepThreshold is 2 and the new Pod-Count would be 4 (pc - 4 < stepThreshold)

Example with multiple queues

-
  label: "example_with_multi_queue"
  deploymentNamespace: ns-1
  deployment: example
  interval: 5m
  queues:
    -
      virtualHost: /hostA
      name: queue_a
      ruleset:
        type: limit
        rules:
          -
            minMessageCount: 0
            podCount: 1
          -
            minMessageCount: 100
            podCount: 2
          -
            minMessageCount: 200
            podCount: 3
    -
      virtualHost: /hostB
      name: queue_b
      ruleset:
        type: linearScale
        rules:
          - factor: 0.005

It is also possible to observe more than one queue per deployment. Each queue has its own ruleset which calculates the new pod-count. The final pod-count is the maximum of all the calculated count of the queue-rulesets.

An example:

Queue-A Message-Count Queue-B Message-Count current Pod-Count new Pod-Count
100 100 cp = 1 2
100 1000 cp = 1 5

GitHub

https://github.com/Max-F-Helm/PodAutoScaler
You might also like...

Taskify is a mobile application used to create and schedule tasks in your TODO list

Taskify is a mobile application used to create and schedule tasks in your TODO list

Taskify is a mobile application used to create and schedule tasks in your TODO list. It is built upon the new Maaterial 3 UI components with the MVVM pattern and the latest Jetpack components.

Jun 9, 2022

Used to generate the template code of GetX framework

Used to generate the template code of GetX framework

Language: English | 中文简体 statement some fast code snippet prompt come from getx-snippets-intelliJ Description install Plugin effect Take a look at the

Jun 13, 2022

Team management service is a production ready and fully tested service that can be used as a template for a microservices development.

Team management service is a production ready and fully tested service that can be used as a template for a microservices development.

team-mgmt-service Description Team management service is a production ready and fully tested service that can be used as a template for a microservice

May 3, 2022

A Kotlin library used to analyse discrete Markov chains, in order to generate plausible sequences

Markov Markov is a Kotlin library used to analyse discrete Markov chains, in order to generate plausible sequences. Using This project is still under

Nov 14, 2021

An open source app which can be used to do basic surveys

An open source app which can be used to do basic surveys

SurveyApp This is an open source app which can be used to do basic surveys. It supports multiple question types. For demo please check the releases pa

Dec 9, 2021

Tweaks - A customizable debug screen to view and edit flags that can be used for development

Tweaks - A customizable debug screen to view and edit flags that can be used for development

A customizable debug screen to view and edit flags that can be used for developm

Mar 4, 2022

Text-array-process - A home assignment used during the recruitment process at Lumera

Rules to implement class={isogram|palindrome|semordnilap} maxlength=INT minl

Jan 27, 2022

Android application compatible with ZX2C4's Pass command line application

Android application compatible with ZX2C4's Pass command line application

Password Store Download Documentation We're in the process of rewriting our documentation from scratch, and the work-in-progress state can be seen her

Jun 23, 2022

This is an example of a simple application with layered software base on clean-architecture as application architecture and mvvm as presentation architecture

This is an example of a simple application with layered software base on clean-architecture as application architecture and mvvm as presentation archi

Jul 2, 2021
Checks if the given grammar is LL(k) for the given k

LL(k) Checker Checks if the given grammar is LL(k) for the given k Grammars Curr

Dec 19, 2021
Map-vs-list-comparator - The project compares the time needed to find a given element in a map vs the time needed to find a given element in a list.

Map vs List Comparator The project compares the time needed to find a given element in a map vs the time needed to find a given element in a list. To

Jan 4, 2022
This repository contains RabbitMQ Protobuf starters with its usage samples for spring-rabbit and spring-cloud-starter-stream-rabbit modules

This repository contains RabbitMQ Protobuf starters with its usage samples for spring-rabbit and spring-cloud-starter-stream-rabbit modules

Nov 29, 2021
Gradle plugin which allows to use typed DSL for generating kubernetes/openshift YAML files

gr8s Gradle plugin which allows using typed DSL for generating kubernetes/openshift YAML files. Based on kuberig Usage import io.github.guai.gr8s.Gene

Jan 3, 2022
Jan 17, 2022
Send Whatsapp Message Without Saving Mobile Number

Send Whatsapp Message Without Saving Mobile Number In this project i created the

Apr 22, 2022
This project aims to provide a solution for finding the right product for a given EAN (European Article Number)

This project aims to provide a solution for finding the right product for a given EAN (European Article Number)

Apr 18, 2022
A backend service that generates a random 32-chars length message and its sha256 hashcode and put them to DB
A backend service that generates a random 32-chars length message and its sha256 hashcode and put them to DB

This is an Android application in which a user can ask the server to generate a random message and its sha256, and ask to obtain this info from the server.

Apr 11, 2022
Kafka Connect JSLT Single Message Transform

This is an implementation of the Kafka Connect SMT interface to offer transformation capabilities using the Schibsted JSLT library.

May 25, 2022
An application that allows the user to update variety of smartphones that are used such as iPhone and Android
An application that allows the user to update variety of smartphones that are used such as iPhone and Android

PhoneApplication An application that allows the user to update variety of smartphones such as iPhone and Android. This application allows users to add

Nov 28, 2021