feat/81 - add CI/CD pipeline to run integration test with CleverSwarm-lite app available in cleverswarm/integration-demo repo #98

Merged

stanislav.hejny merged 159 commits from feat/81 into develop 2025-11-06 16:39:10 +00:00

Conversation 0 Commits 159 Files changed 125 +13639 -643

stanislav.hejny commented 2025-11-06 16:38:15 +00:00

Member

Copy link

No description provided.

stanislav.hejny added 159 commits 2025-11-06 16:38:15 +00:00

feat/#81: Try to execute integration test within CI/CD pipeline bfeafb253e

feat/#83 - add cleverswarm dependency and extra debug statements 4e801b4728

feat/#83 - fix the s3 download c1749899a9

feat/#83 - fix the input parsing for dictionary input 23ab3ee0c1

feat/#83 - fix the replica count read command 3957a741ea

Wait keda to ready before deploying clevermicro ... 977bc5ac0d

Add `--wait` flag to helm install command when installing keda, this ensures the keda is ready for the scaling test.

feat/#83 - add KEDA sclaingObj

feat/#83 - add KEDA scalingObj and apply

fix: add swarm task id and service id for otdemo deployment

feat/#83 - add extra log prints

feat/#83 - add extra log prints

feat/#83 - add extra log prints

feat/#83 - add extra log prints

feat/#83 - add extra log prints 2

feat/#83 - synchronize docker image

feat/#83 - synchronize docker image 2

feat/#83 - synchronize docker image 3

feat/#83: add extra wait for scale down, don't print all logs

fix: add curl to update consul setting for testing the scaling down

feat/#83: logs on cleanup, reduce the interval otdemo sends backpressure notifications

feat/#83: remove the temp failure override for the scaling test

feat/#83: remove the temp failure override for the scaling test

feat/#83: update scaling test script

feat/#83: update scaling test script 2

feat/#83: diagnostic print of frontend log after test failsp

feat/#83: diagnostic print of frontend log after test fails 2

feat/#83: fix the 'cannot schedule new futures after shutdown' error

feat/#83: parametrize docker tag to ensure latest build is used

feat/#83: fix typo in the action script

feat/#83: fix the FastAPI and AMQ adapter to work together

feat/#83: remove unnecessary pod restart from action script

feat/#83: extend the scaling test duration attempting to capture  more replicas being created

feat/#83: print Prometheus metrics that triggers scaling

feat/#83: comment and cleanup scripts

ci: merge pipelines into one main pipeline 57eb5ac6c1

ci: lower coverage rate requirement to 80 so I can debug the rest of the pipeline 3c8ef2a793

ci: make endpoint test fail the pipeline, increase scaling test waiting time af0d25ad45

ci: add a bit waiting for the scaling test 3273ba6a94

ci: debug /health endpoint 2564723764

ci: add warm up before integration test 33e55a80d7

ci: build one otdemo image and use that for live testing 7f3f6b35c7

ci: match the docker image tag between build and testing step 8289b9b4fa

ci: change scaling script to use loose condition when waiting for scale down 77637fbadc

ci: reduce the OTDemo app CPU limit and request to allow pod placement 10483008c5

ci: use shared action for deploying clevermicro 83ac5ac467

ci: make otdemo integration test a shared action d971d3f3a6

ci: remove unnecessary scripts ed5d6cafe0

ci: use minio as s3 replacement 0adffb9b69

feat/#83: add NodePort to expose Prometheus API endpoint 8398568ac2

ci: remove setup python cache 4e1c34d7c6

ci: switched to the shared action 09ce3f9c5b

Add test cases for S3 handler, there's still few failures b459c9ed42

Merge remote-tracking branch 'origin/feat/81-shared-actions' into feat/81 4411e90755

feat/#83: fix failing tests b0ed17b6ea

feat/#83: fix failing tests 2 48d0a53895

ci: test the python setup for CSwarm - 1 350156ad52

ci: test the python setup for CSwarm - 2 396c9af2ba

ci: test the python setup for CSwarm - 2 0a8347f1ca

ci: test the python setup for CSwarm - 3 282e7a489a

ci: fix typo in setup for CSwarm - 4 12f34f7f03

ci: add Docker login in setup for CSwarm - 5 f38b2dab42

ci: add registry inputs 58eb26a3d9

fix: use default namespace c5c445eb5e

fix: use default namespace 5841f7e941

ci: rename the image to see if this helps issue with downloading ... 6feeb64af8

# Conflicts:
#	.forgejo/workflows/main.yaml

feat/#83: use the CleverSwarm tests for endpoint testing, add KEDA for frontend scaling as well fcfb6218e1

feat/#83: debug address http://localhost:30080/cleverswarm/api/v0/login not reachable, take 1 6b2c4218b2

feat/#83: debug address http://localhost:30080/cleverswarm/api/v0/login not reachable, take 2 929ecd30e8

feat/#83: debug address http://localhost:30080/cleverswarm/api/v0/login not reachable, take 3 1367410672

ci: add imagePullSecrets to cleverswarm frontend and backend deployment manifests d03a000a45

ci: fix namespace for otdemo test 669c145019

fix: update otdemo docker entrypoint to forward SIGTERM 010acae711

ci: disable otdemo test 8a9b08f709

feat/#83: Add more debug to action, modify local env to match the new shared actions after refactoring af08967a25

fix: use default namespace 4750466880

feat/#83: Add 2min sleep before test start to give time integration-demo app to start cdf29bd303

feat/#83: use newest integration-demo image with updated rabbitmq URL 224f2c9abb

feat/#83: make adapter use RABBITMQ_URL env variable, modify pipline to allow wheel publish from this branch e0fd7e96a9

feat/#83: make adapter use RABBITMQ_URL env variable, modify pipline to allow wheel publish from this branch, take 2 7c63ea64ba

feat/#83: make adapter use RABBITMQ_URL env variable, modify pipline to allow wheel publish from this branch, take 3 a31a2b054e

feat/#83: Use the latest image for CleverSwarm integration-demo that should fix RabbitMQ connection problem ce5cc45a57

feat/#83: add env var, use latest integration-demo image 92cfff7488

feat/#83: add env var, take 2 c8ae9c79c9

feat/#83: add RabbitMQ stats printout, take 1 01346c532a

feat/#83: add RabbitMQ stats printout, take 2 2ee5e5c28b

feat/#83: add RabbitMQ UI NodePort forwarding, take 1 8a7fd1451e

feat/#83: add RabbitMQ UI NodePort forwarding, take 2 b29485a81a

feat/#83: add RabbitMQ UI NodePort forwarding, take 3 d0bc9bf3b7

feat/#83: print RabbitMQ stats, take 1 b89e3d610a

feat/#83: Add extra debug to identify the config file source 390ddddc8f

feat/#83: Add extra debug to identify the config file source 16c405217e

feat/#83: Use int-demo img with latest: Add extra debug to identify the config file source e3ec3f1a45

feat/#83: add printout of properties 7fbad3b246

feat/#83: add dump of properties file 8f03c328d9

feat/#83: config changes to work around performance and past errors in setup 9c53edd8e5

feat/#83: Add MinIO support, make scaling test run before endpoint compatibility test 4c0cc0d1da

feat/#83: fix empty string conversion 6f3387cba5

feat/#83: Use the latest integration-demo image with AMQ Adapter Lib v0.2.56 e965d796fa

feat/#83: Use the latest integration-demo image with AMQ Adapter Lib v0.2.56 and minio module installed ee8610196e

feat/#83: Fix the MinIO endpoint to remove redundant protocol spec 8b14b9b425

ci: copy test file to correct forlder 20aedb9b46

refactor: ci 290367c56f

feat/#83: Fix the wrong label in scaling test 4999f9a8cf

feat/#83: Fix the wrong label in scaling test, take 2 7ee61e67df

feat/#83: Add pydantic type instantiation 5a6261a8c1

feat/#83: Add pydantic type instantiation, fix unit test d27bdd20cf

feat/#83: Add pydantic type instantiation, fix unit test 0871710898

feat/#83: Use integartion-demo img with pydantic type instantiation 5feadcb5a4

feat/#83: Use env variable to configure LOAD_AS_BACKPRESSURE feature e9655e0d77

feat/#83: Use env variable to configure LOAD_AS_BACKPRESSURE feature, build bec254bb40

feat/#83: fix KEDA HPA's references b588f4d899

fix: keda prometheus endpoint url ... 76cb2cae70

keda is running in namespace `keda`, while others are running in the default namespace, thus the full url is required for keda to access prometheus.

feat/#83: Provide SWARM_SERVICE_ID env variable with setting matching the KEDA reference 65dc7eeba7

feat/#83: Provide SWARM_SERVICE_ID env variable with setting matching the KEDA reference, fix compile error 5727309e06

feat/#83: Do not throw exception when pydantic type can't be instantiated, return None instead dd302ecdd6

feat/#83: Do not throw exception when pydantic type can't be instantiated, return None instead 4da0987609

feat/#83: use latest library and image dbda039473

feat/#83: use latest library and image 2 a95891fb08

ci: rename otdemo test to internal integration test f8f3f56816

ci: update scaling test script 42d9eb48c3

feat/#83: create a static version of backpressure reporter API 04fb04d7a5

feat/#83: use latest library, synch config data beb0def7ef

ci: add with s3 variant and update scaling test fd71a1d10e

ci: add test streaming script 3d26a8cc1e

ci: remove resource limit for demo service 945d0e0165

debug: kind cluster setup error d10adace7d

ci: add debug log output c910f78b90

ci: add http endpoint test script 5b2af66a32

ci: remove minio and use s3 to save some resources ... 3b2906ea94

also free up more resources by removing services that we don't need

feat/#83: use latest integration-demo, switch to use S3 again instead of MinIO 9c65125309

feat/#83: use latest integration-demo, switch to use S3 again instead of MinIO,take 2 58857afc50

feat/#83: use latest integration-demo, switch to use S3 again instead of MinIO,take 3 fd2bf2ecd0

ci: update amq adapter version f83984cae3

Merge branch 'feat/90' into feat/81 357dc87910

feat/#93: fix bug where REMOVE_ROUTE event on scaling down invalidates routes to all replicas 0d390876a0

feat/#93: fix failing tests after refactor 4681cd287f

feat/#93: do not run integration test when I need to build a wheel file 0a58726825

feat/#93: do not run integration test when I need to build a wheel file e903ade5ad

feat/#93: do not run integration test when I need to build a wheel file, fix build condition 5c310c1573

feat/#93: do not run integration test when I need to build a wheel file, fix build condition f74f0fbf63

feat/#93: do not run integration test when I need to build a wheel file, fix build condition 46e80988ac

feat/#93: do not run integration test when I need to build a wheel file, fix build condition 7b8b443222

feat/#83: use the amq-router that fixes 503 error cd53c04113

Merge branch 'feat/81' into feat/93 5f67332672

Merge pull request 'feat/93' (#94) from feat/93 into feat/81 ... 0e2651915d

Reviewed-on: #94

refactor: ci ... fb9abe1ace

# Conflicts:
#	.forgejo/actions/integration-test-with-otdemo/action.yaml
#	.forgejo/workflows/main.yaml

ci: refactor 6dd98adaac

ci: increase wait time for cleverswarm startup f091e46e85

ci: describe the correct scaler 7eff16081c

Merge branch 'feat/81-fixes' into feat/81 ... 259b8b3dde

# Conflicts:
#	.forgejo/actions/integration-test-with-cleverswarm/cleverswarm-backend-deployment.yaml
#	.forgejo/actions/integration-test-with-cleverswarm/cleverswarm-frontend-deployment.yaml
#	.forgejo/actions/integration-test-with-cleverswarm/keda-cleverswarm-backend-scaling.yaml
#	.forgejo/actions/integration-test-with-cleverswarm/keda-cleverswarm-frontend-scaling.yaml
#	.forgejo/actions/integration-test-with-otdemo/keda-otdemo-backend-scaling.yaml
#	.forgejo/actions/integration-test-with-otdemo/otdemo-backend-deployment.yaml
#	.forgejo/actions/integration-test-with-otdemo/otdemo-backend-service.yaml
#	.forgejo/actions/integration-test-with-otdemo/otdemo-frontend-deployment.yaml
#	.forgejo/actions/integration-test-with-otdemo/otdemo-frontend-service.yaml
#	.forgejo/actions/integration-test-with-otdemo/test_scaling.py
#	.forgejo/actions/integration-tests/cleverswarm/action.yaml
#	.forgejo/actions/integration-tests/cleverswarm/test_scaling.py
#	.forgejo/actions/integration-tests/otdemo/k8s-prometheus-forward.yaml
#	.forgejo/actions/integration-tests/otdemo/keda-otdemo-backend-scaling.yaml
#	.forgejo/actions/integration-tests/otdemo/otdemo-backend-deployment.yaml
#	.forgejo/actions/integration-tests/otdemo/otdemo-backend-service.yaml
#	.forgejo/actions/integration-tests/otdemo/otdemo-frontend-deployment.yaml
#	.forgejo/actions/integration-tests/otdemo/otdemo-frontend-service.yaml
#	.forgejo/actions/integration-tests/otdemo/test_demo_endpoints.py
#	.forgejo/actions/integration-tests/otdemo/test_scaling.py
#	.forgejo/actions/internal-intergration-test-with-demo-apps/keda-otdemo-backend-scaling.yaml
#	.forgejo/actions/internal-intergration-test-with-demo-apps/otdemo-backend-deployment.yaml
#	.forgejo/actions/internal-intergration-test-with-demo-apps/otdemo-backend-service.yaml
#	.forgejo/actions/internal-intergration-test-with-demo-apps/otdemo-frontend-deployment.yaml
#	.forgejo/actions/internal-intergration-test-with-demo-apps/otdemo-frontend-service.yaml
#	.forgejo/actions/publish-python-library/action.yaml
#	.forgejo/actions/unit-tests/action.yaml
#	.forgejo/workflows/main.yaml
#	.gitignore
#	amqp/adapter/consul_global_id_generator.py
#	amqp/config/amq_configuration.py
#	pyproject.toml

feat/#83: update image versions c164cbfc3a

feat/#83: correct path tp ScaledObject manifest a05edd0da9

feat/#83: Attempt to fix "Insufficient storage error" on the target pod c57fea4cf3

feat/#83: Attempt to fix "Insufficient storage error" on the target pod, take 2 9e26d5a455

feat/#83: Attempt to fix "Insufficient storage error" on the target pod, take 3 8f4edfa0e7

feat/#83: Attempt to fix "Insufficient storage error" on the target pod, take 4 fbb8672919

feat/#83: Fix typo causng file not being found bbdc265bd6

feat/#83: warm up path before running tests 88d736adc6

feat/#83: build amq 0.2.65 8ac4b5eac0

feat/#83: build amq 0.2.65 efa2066045

feat/#83: build amq 0.2.65 06b0710cd6

feat/#83: attempt fixing build yaml error to build amq 0.2.65, take 1 6bce12b911

feat/#83: use image with amq 0.2.65 and extra debug, take 1 f9e73b64b6

feat/#83: build amq 0.2.66 and extra debug, take 1 3fd0f10fc3

feat/#83: use demo image with amq 0.2.66 and extra debug, take 1 bf7aee42a4

feat/#83: use demo image with amq 0.2.66 and /tmp as job storage 362a888d19

Merge pull request 'feat/81b merge the fixes to pipeline back to feature branch' (#96) from feat/81b into feat/81 ... 676ed2f71c

Reviewed-on: #96

feat/#83: use demo image with BASE_DIR change to /tmp f339cab45b

feat/#83: use demo image with BASE_DIR as base to JOBS_FILE_PATH 21af912ac6

feat/#83: use demo image with BASE_DIR as base to JOBS_FILE_PATH 69001f72b3

Merge pull request 'feat/81b' (#97) from feat/81b into feat/81 ... 9c7cf63a9c

Reviewed-on: #97

feat/#83: cleanup obsolete actions, add better loging to find_route() method 46b60c6b7e

feat/#83: build/publish amq 0.2.67 3c0e249724

feat/#83: build/publish amq 0.2.68 75bd419ef7

feat/#83: cleanup tool, build/publish amq 0.2.69 with more debug prints fe2b626ebc

feat/#83: fix header serialization, build/publish amq 0.2.70 8d1ad87cb5

feat/#83: try the tests with fixed integration-demo and bigger timeouts d760e48358

feat/#83: try the tests with fixed integration-demo and bigger timeouts, take 2 2f5d6dd515

feat/#83: run the endpoint test without replication, take 1 800c439642

feat/#83: run the endpoint test without replication, take 2 a81519208d

feat/#83: run the endpoint test without replication, take 3 2dbb25cb67

feat/#83: run the endpoint test without replication, take 4 def3571b19

ci: fix secrets name cbc9008af4

feat/#83: run the endpoint test without replication, take 5 00ea45bcf6

feat/#83: run the endpoint test without replication, take 6 f3f89df42d

feat/#83: run the endpoint test without replication, take 7 60358218db

feat/#83: run the endpoint test without replication, add test2 for large files, take 8 f729a4fd72

stanislav.hejny self-assigned this 2025-11-06 16:38:20 +00:00

stanislav.hejny added this to the V.01 milestone 2025-11-06 16:38:26 +00:00

stanislav.hejny added this to the Sprint Nov 1st to 14th project 2025-11-06 16:38:34 +00:00

stanislav.hejny merged commit 056a46e1ed into develop 2025-11-06 16:39:10 +00:00

stanislav.hejny referenced this pull request from a commit 2025-11-06 16:39:11 +00:00

Merge pull request 'feat/81 - add CI/CD pipeline to run integration test with CleverSwarm-lite app available in cleverswarm/integration-demo repo' (#98) from feat/81 into develop

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

  

Blocked

A ticket in a blocked state and unable to complete until some other task is completed first.

  

Bounty

$100

A bounty of $100 for any open-source contributor who provides a MR that solves this issue

  

Bounty

$1000

A bounty of $1000 for any open-source contributor who provides a MR that solves this issue

  

Bounty

$10000

A bounty of $10000 for any open-source contributor who provides a MR that solves this issue

  

Bounty

$20

A bounty of $20 for any open-source contributor who provides a MR that solves this issue

  

Bounty

$2000

A bounty of $2000 for any open-source contributor who provides a MR that solves this issue

  

Bounty

$250

A bounty of $250 for any open-source contributor who provides a MR that solves this issue

  

Bounty

$50

A bounty of $50 for any open-source contributor who provides a MR that solves this issue

  

Bounty

$500

A bounty of $500 for any open-source contributor who provides a MR that solves this issue

  

Bounty

$5000

A bounty of $5000 for any open-source contributor who provides a MR that solves this issue

  

Bounty

$750

A bounty of $750 for any open-source contributor who provides a MR that solves this issue

  

MoSCoW

Could have

Could have feature in order to satisfy the epic/legendary.

  

MoSCoW

Must have

Must have feature in order to satisfy the epic/legendary.

  

MoSCoW

Should have

Should have feature in order to satisfy the epic/legendary.

  

Needs feedback

There are questions in the ticket that can not be completed until the project owner provides clarity.

  

Points

1

1 man-hours worth of work for an expert with no learning curve.

  

Points

13

13 man-hours worth of work for an expert with no learning curve.

  

Points

2

2 man-hours worth of work for an expert with no learning curve.

  

Points

21

21 man-hours worth of work for an expert with no learning curve.

  

Points

3

3 man-hours worth of work for an expert with no learning curve.

  

Points

34

34 man-hours worth of work for an expert with no learning curve.

  

Points

5

5 man-hours worth of work for an expert with no learning curve.

  

Points

55

55 man-hours worth of work for an expert with no learning curve.

  

Points

8

8 man-hours worth of work for an expert with no learning curve.

  

Points

88

88 man-hours worth of work for an expert with no learning curve.

  

Priority

Backlog

This ticket has backlogged priority and is not to be worked on yet

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Signed-off: Owner

When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.

  

Signed-off: Scrum Master

When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.

  

Signed-off: Tech Lead

When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.

  

Spike

A ticket for learning a tool or technology that is needed to be able to do future planning and design.

  

State

Completed

The ticket has been fully implemented, completed, and merged with the source code. This label should only be applied once a ticket is closed.

  

State

Duplicate

A ticket that represents the same content as an existing ticket.

  

State

In Progress

A ticket that is actively being developed.

  

State

In Review

A ticket that has had some code completed to implement but is waiting to pass peer review and is not yet merged in.

  

State

Paused

This ticket's work started but wasn't finished. It's on hold (likely in a feature branch) and will be resumed later, either due to a blocker or a delay.

  

State

Unverified

All new tickets start in this state. A developer may set it to show the ticket is unverified. This means we haven't agreed to work on it. It will either move to a verified state or be closed as wontdo.

  

State

Verified

The issue has been verified by a developer as legitimate. It will be worked on and verified tickets are now considered part of the backlog.

  

State

Wont Do

This ticket has been decided it wont be done. This may mean the bug has been determined to not be real (cant verify) or the feature is one we have decided we dont want to adopt.

  

Type

Bug

Something that doesnt work as intended.

  

Type

Discussion

Anytime a ticket represents a discussion about a subject and doesnt fall into one of the other categories.

  

Type

Documentation

An error or improvement needed in the documentation.

  

Type

Epic

Any first tier epic. That is, an epic which contains only issues as children and will not have sub-epics.

  

Type

Feature

Some new functionality not present.

  

Type

Legendary

A type of Epic which will contain other Epics.

  

Type

Support

Someone needs help using the project.

  

Type

Task

A generic task that doesnt fit into the other type categories.

  

Type

Testing

Work exclusively focusing on fixing or expanding testing.

No labels

Blocked

Bounty

$100

Bounty

$1000

Bounty

$10000

Bounty

$20

Bounty

$2000

Bounty

$250

Bounty

$50

Bounty

$500

Bounty

$5000

Bounty

$750

MoSCoW

Could have

MoSCoW

Must have

MoSCoW

Should have

Needs feedback

Points

1

Points

13

Points

2

Points

21

Points

3

Points

34

Points

5

Points

55

Points

8

Points

88

Priority

Backlog

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Signed-off: Owner

Signed-off: Scrum Master

Signed-off: Tech Lead

Spike

State

Completed

State

Duplicate

State

In Progress

State

In Review

State

Paused

State

Unverified

State

Verified

State

Wont Do

Type

Bug

Type

Discussion

Type

Documentation

Type

Epic

Type

Feature

Type

Legendary

Type

Support

Type

Task

Type

Testing

Milestone

Clear milestone

No items

No milestone

V.01

Projects

Clear projects

No items

No project

Sprint Nov 1st to 14th

Assignees

Clear assignees

No assignees

stanislav.hejny

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

clevermicro/amq-adapter-python!98

No description provided.