-
Notifications
You must be signed in to change notification settings - Fork 241
[AURON #1264] Add BatchExecCalc shadow for Flink BATCH mode Calc - #2516
Draft
argoyal2212 wants to merge 1 commit into
Draft
[AURON #1264] Add BatchExecCalc shadow for Flink BATCH mode Calc #2516argoyal2212 wants to merge 1 commit into
argoyal2212 wants to merge 1 commit into
Conversation
argoyal2212
force-pushed
the
argoyal/flink-batch-calc-shadow
branch
from
September 8, 2026 20:57
6f1d8de to
4efc256
Compare
Mirrors the existing StreamExecCalc shadow so native execution also covers BATCH mode's Calc operator, reusing NativePlanFusionBuilder and FlinkAuronCalcOperator unchanged. Fusion codegen and source fusion are out of scope for this first step. Relates to apache#1264
argoyal2212
force-pushed
the
argoyal/flink-batch-calc-shadow
branch
from
September 8, 2026 20:59
4efc256 to
ee603ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
Which issue does this PR close?
TBD
Rationale for this change
Flink's BATCH mode currently has no native execution:
DefaultPlannerFactoryroutes it to the stock, unmodifiedBatchPlanner.NativePlanFusionBuilderandFlinkAuronCalcOperatorare already engine-agnostic (no streaming-specific types), so the smallest correct first step is aBatchExecCalcshadow reusing both, mirroring the existingStreamExecCalcshadow.What changes are included in this PR?
org.apache.flink.table.planner.plan.nodes.exec.batch.BatchExecCalcinauron-flink-planner, matching Flink 1.18.1's stock constructor signature exactly.NativePlanFusionBuilder.buildNativeCalcPlan, falling back to Flink's codegen Calc (or throwing if fallback is disabled), same as streaming.AuronOperatorFusionProcessor);supportFusionCodegen()returnsfalse.@ExecNodeMetadata/@JsonCreator, matching stock batchBatchExecCalcat this Flink version (compiled-plan restoration is streaming-only here).shadowed-flink-execnodes.txt; added matching shade<exclude>inauron-flink-assembly/pom.xml.No changes to any streaming file or to
DefaultPlannerFactory.Are there any user-facing changes?
Yes: BATCH queries now get native execution for Calc (projection/filter) when
auron-flink-assemblyis ahead of stockflink-table-planneron the classpath. Falls back to Flink's codegen Calc when not convertible, matching existing streaming behavior.How was this patch tested?
ShadowedExecNodeRegistryTestandAssemblyJarStructureITpass with the new class registered and shaded correctly.@ExecNodeMetadata/@JsonCreatoragainst the realflink-table-planner_2.12-1.18.1.jar.ProjectExec/FFIReaderin task logs).Draft while a dedicated
BatchExecCalcTest(mirroringStreamExecCalcTest) is added.Was this patch authored or co-authored using generative AI tooling?
ASF guidance: https://www.apache.org/legal/generative-tooling.html