Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

SQL Dialect Extension API for Column Lineage #677

long-xjh started this conversation in Ideas
Discussion options

Problem

Altimate Code's lineage engine supports many SQL dialects, but users working with dialect variants (MaxCompute, cloud-specific
extensions, etc.) have no way to customize dialect behavior without forking the codebase.

Concretely, MaxCompute SQL parsed as hive silently drops lineage edges for functions whose signatures diverge from Hive —
notably DATEADD, NVL, IF, ARRAY_SORT, FILTER, TRANSFORM, FROM_JSON. Real pipeline queries produce incomplete
lineage with no warning.

Proposed Solution: Dialect Extension API

A public API that lets users register custom dialect configurations via YAML/JSON without modifying Altimate Code internals:

  1. Dialect inheritance — define a new dialect that extends an existing one (e.g., maxcompute extends hive) with only the
    overrides needed
  2. Custom function signatures — register argument names and lineage behavior for dialect-specific functions
  3. Function-to-function mappings — define how dialect-specific functions map to other dialects for transpilation

Example config (.altimate/dialects/maxcompute.yaml):

name: maxcompute
extends: hive
overrides:
 - DATEADD
 - NVL
 - IF
 - ARRAY_SORT
 - FILTER
 - TRANSFORM
 - FROM_JSON
Why This Benefits Everyone
- MaxCompute and other variant dialects get accurate lineage without core code changes 
- New dialects become self-serve — no PR needed for dialect support
- The existing dialect abstraction is already there; this surfaces it as a public API 
- Community can share dialect configs (similar to how dbt packages share profiles) 
 
References 
 
- Current dialect handling: packages/opencode/src/altimate/native/dbt/lineage.ts 
- altimate-core dialect engine: @altimateai/altimate-core
 
Happy to contribute a prototype with real MaxCompute SQL test cases.
You must be logged in to vote

Replies: 1 comment

Comment options

@long-xjh Please contribute a prototype. We will take it from there.

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Ideas
Labels
None yet

AltStyle によって変換されたページ (->オリジナル) /