[Proposal] Consolidate CI Workflows Using Matrix Strategy #1696

Proposers

@tuhaihe

Proposal Status

Under Discussion

Abstract

This proposal suggests consolidating our current four separate CI workflow files (Rocky 8, Rocky 9, Ubuntu 22.04, Ubuntu 24.04 - coming Rocky 10) into two unified workflows using GitHub Actions matrix strategy - one for RPM-based distributions and one for DEB-based distributions.

Motivation

This consolidation aligns with GitHub Actions best practices and will make our CI infrastructure more maintainable and scalable without increasing PR validation time. The matrix strategy is specifically designed for this use case.

This consolidation is now feasible because our project has evolved to use unified build scripts (devops/build/automation/cloudberry/scripts/) that work consistently across different OS versions, rather than requiring OS-specific configurations. The workflows primarily differ only in the container images used, making them ideal candidates for matrix strategy.

Implementation

Current State

We currently maintain four separate workflow files:

Issues with Current Approach

1. Code Duplication: The Rocky 8/9 workflows are nearly identical (~1,900 lines each), as are the Ubuntu 22.04/24.04 workflows (~1,900 lines each). This leads to:

   • Maintenance burden when updating test configurations
   • Risk of inconsistencies between versions
   • Copy-paste errors

2. Inconsistent PR Coverage: Rocky 8 and Ubuntu 24.04 are not tested during PR reviews, potentially missing compatibility issues until after merge.

3. Scalability: Adding more newer OS version support would require creating yet another ~1,000 line workflow file.

Proposed Solution

Consolidate into two matrix-based workflows:

1. build-cloudberry-rpm.yml (RPM-based distributions)

name: Apache Cloudberry RPM Build
on:
 push:
 branches: [main, REL_2_STABLE]
 pull_request:
 branches: [main, REL_2_STABLE]
 types: [opened, synchronize, reopened, edited]
 workflow_dispatch:
jobs:
 build:
 strategy:
 matrix:
 os_version: [rocky8, rocky9]
 # Future: rocky10 can be added here
 container:
 image: apache/incubator-cloudberry:cbdb-build-${{ matrix.os_version }}-latest

2. build-cloudberry-deb.yml (DEB-based distributions)

name: Apache Cloudberry DEB Build
on:
 push:
 branches: [main, REL_2_STABLE]
 pull_request:
 branches: [main, REL_2_STABLE]
 types: [opened, synchronize, reopened, edited]
 workflow_dispatch:
jobs:
 build:
 strategy:
 matrix:
 os_version: [ubuntu22.04, ubuntu24.04]
 container:
 image: apache/incubator-cloudberry:cbdb-build-${{ matrix.os_version }}-latest

Benefits

1. No Increase in PR CI Time. GitHub Actions matrix jobs run in parallel, not sequentially. Same total time, more coverage

2. Reduced Maintenance Burden. When updating test configurations, changes only need to be made once per package type (RPM/DEB) instead of twice.

3. Better Test Coverage. All supported OS versions will be tested on every PR, catching compatibility issues earlier in the development cycle.

4. Easier to Scale.

Adding Rocky 10 support becomes trivial:

matrix:
 os_version: [rocky8, rocky9, rocky10] # Just add one line

5. Consistency Guaranteed. Since all OS versions use the same workflow logic, they're guaranteed to run identical tests with identical configurations (except for OS-specific differences).

Rollout/Adoption Plan

Love to hear your thoughts and feedback!

Are you willing to submit a PR?

• Yes I am willing to submit a PR!