Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

[TT-15825] [BE] Address inconsistencies with use of Policy identifiers #7424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shults wants to merge 3 commits into master
base: master
Choose a base branch
Loading
from TT-15825-be-address-inconsistencies-with-use-of-policy-identifiers

Conversation

Copy link
Contributor

@shults shults commented Oct 8, 2025
edited by github-actions bot
Loading

User description

TT-15825
Summary [BE] Address inconsistencies with use of Policy identifiers
Type Bug Bug
Status In Dev
Points N/A
Labels 2025_long_tail, AI-Complexity-Medium, AI-Priority-High, codilime_refined, customer_bug, jira_escalated, stability_theme_auth

Description

Related Issue

Motivation and Context

How This Has Been Tested

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring or add test (improvements in base code or adds test coverage to functionality)

Checklist

  • I ensured that the documentation is up to date
  • I explained why this PR updates go.mod in detail with reasoning why it's required
  • I would like a code coverage CI quality gate exception and have explained why

PR Type

Bug fix, Tests, Documentation


Description

  • Remove explicit policy ID toggle usage

  • Standardize policy ID to policy.ID

  • Update RPC/Dashboard loaders signatures

  • Adjust tests for new behavior


Diagram Walkthrough

flowchart LR
 Config["Config: deprecate allow_explicit_policy_id"] -- "not used" --> Loaders["Policy loaders"]
 Loaders["Policy loaders"] -- "use policy.ID only" --> Dashboard["LoadPoliciesFromDashboard"]
 Loaders -- "use policy.ID only" --> RPC["LoadPoliciesFromRPC/parsePoliciesFromRPC"]
 API["handleAddOrUpdatePolicy"] -- "backfill ID from MID" --> Filesave["Save policy file"]
 Tests["Tests updated"] -- "drop allowExplicit param" --> Dashboard
 Tests -- "expect IDs via policy.ID" --> RPC
 Backup["RPC backup handlers"] -- "parsePoliciesFromRPC()" --> RPC
Loading

File Walkthrough

Relevant files
Documentation
config.go
Deprecate unused AllowExplicitPolicyID config

config/config.go

  • Mark AllowExplicitPolicyID as deprecated.
  • Note it's unused in codebase.
+2/-0
Bug fix
api.go
Backfill missing policy ID from MID on save

gateway/api.go

  • Default newPol.ID to newPol.MID.Hex() if empty.
  • Ensure saved filename uses resolved policy ID.
+4/-0
Enhancement
policy.go
Standardize ID handling and simplify loader APIs

gateway/policy.go

  • Remove allowExplicit parameter from loaders/parsers.
  • Use policy.ID as key; drop MID fallback.
  • Adjust duplicate detection to use policy.ID.
  • Update retry path to new signatures.
+8/-18
rpc_backup_handlers.go
Update RPC backup to new parsing/decrypt API

gateway/rpc_backup_handlers.go

  • Call parsePoliciesFromRPC without allowExplicit.
  • Use updated crypto decrypt signature.
+2/-2
server.go
Remove allowExplicit parameter from sync paths

gateway/server.go

  • Call loaders without AllowExplicitPolicyID.
  • Keep source selection logic intact.
+2/-2
Tests
policy_test.go
Align tests with standardized policy ID handling

gateway/policy_test.go

  • Update tests to new loader signatures.
  • Adjust expectations to lookup by policy.ID.
  • Remove dependency on config AllowExplicitPolicyID.
+28/-37

probelabs[bot] reacted with thumbs up emoji
Copy link
Member

buger commented Oct 8, 2025

I'm a bot and I 👍 this PR title. 🤖

Copy link
Contributor

github-actions bot commented Oct 8, 2025

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
Recommended focus areas for review

Possible Issue

When creating or updating a policy, if the incoming policy has an empty ID, the code sets newPol.ID to newPol.MID.Hex(). Ensure MID is always populated for new policies; otherwise MID may be zero-value and .Hex() could yield an empty or invalid ID, leading to filename collisions or unexpected behavior.

if newPol.ID == "" {
	newPol.ID = newPol.MID.Hex()
}
Duplicate Handling

Policies fetched from the Dashboard now strictly use p.ID as the key and skip duplicates. Validate that upstream always sets ID (and uniquely) after deprecating explicit/Mongo ID switching; otherwise blank or reused IDs will cause policies to be dropped silently.

	if _, ok := policies[p.ID]; ok {
		log.WithFields(logrus.Fields{
			"prefix": "policy",
			"policyID": p.ID,
			"OrgID": p.OrgID,
		}).Warning("--> Skipping policy, new item has a duplicate ID!")
		continue
	}
	policies[p.ID] = p.ToRegularPolicy()
}
Crypto API Change

crypto.Decrypt is now called with secret (string/bytes change). Confirm the function signature expects the provided type; mismatches could cause decryption failure and policy-loading errors in backup recovery.

secret := crypto.GetPaddedString(gw.GetConfig().Secret)
cryptoText, err := store.GetKey(checkKey)
listAsString := crypto.Decrypt(secret, cryptoText)
if err != nil {
	return nil, errors.New("[RPC] --> Failed to get node policy backup (" + checkKey + "): " + err.Error())
}
if policies, err := parsePoliciesFromRPC(listAsString); err != nil {
	log.WithFields(logrus.Fields{

Copy link
Contributor

github-actions bot commented Oct 8, 2025
edited
Loading

API Changes

--- prev.txt	2025年10月14日 12:44:46.085105890 +0000
+++ current.txt	2025年10月14日 12:44:36.618225843 +0000
@@ -6879,6 +6879,8 @@
 	// If you set this value to `true`, then the id parameter in a stored policy (or imported policy using the Dashboard API), will be used instead of the internal ID.
 	//
 	// This option should only be used when moving an installation to a new database.
+	//
+	// Deprecated. Is not used in codebase.
 	AllowExplicitPolicyID bool `json:"allow_explicit_policy_id"`
 	// This option only applies in OSS deployment when the `policies.policy_source` is either set
 	// to `file` or an empty string. If `policies.policy_path` is set, then Tyk will load policies
@@ -10009,11 +10011,11 @@
 
 func (gw *Gateway) LoadDefinitionsFromRPCBackup() ([]*APISpec, error)
 
-func (gw *Gateway) LoadPoliciesFromDashboard(endpoint, secret string, allowExplicit bool) (map[string]user.Policy, error)
+func (gw *Gateway) LoadPoliciesFromDashboard(endpoint, secret string) (map[string]user.Policy, error)
 LoadPoliciesFromDashboard will connect and download Policies from a Tyk
 Dashboard instance.
 
-func (gw *Gateway) LoadPoliciesFromRPC(store RPCDataLoader, orgId string, allowExplicit bool) (map[string]user.Policy, error)
+func (gw *Gateway) LoadPoliciesFromRPC(store RPCDataLoader, orgId string) (map[string]user.Policy, error)
 
 func (gw *Gateway) LoadPoliciesFromRPCBackup() (map[string]user.Policy, error)
 

Copy link
Contributor

github-actions bot commented Oct 8, 2025

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion Impact
Possible issue
Normalize empty IDs from dashboard

Handle empty p.ID values to avoid inserting entries under an empty key and silently
dropping multiple policies. Default empty IDs to p.MID.Hex() (or another stable
identifier) before deduplication.

gateway/policy.go [184-194]

 for _, p := range list.Message {
+	if p.ID == "" {
+		if hex := p.MID.Hex(); hex != "" {
+			p.ID = hex
+		} else {
+			p.ID = persistentmodel.NewObjectID().Hex()
+		}
+	}
 	if _, ok := policies[p.ID]; ok {
 		log.WithFields(logrus.Fields{
 			"prefix": "policy",
 			"policyID": p.ID,
 			"OrgID": p.OrgID,
 		}).Warning("--> Skipping policy, new item has a duplicate ID!")
 		continue
 	}
 	policies[p.ID] = p.ToRegularPolicy()
 }
Suggestion importance[1-10]: 8

__

Why: Addresses potential empty p.ID leading to empty-key map entries and lost policies; aligns with prior behavior using MID. High impact on correctness if IDs can be empty.

Medium
Derive IDs for RPC policies

Prevent empty-key insertions when parsing policies from RPC by deriving an ID if
p.ID is empty. Use p.MID.Hex() or generate a new ID to ensure map integrity and
consistent lookups.

gateway/policy.go [208-210]

 for _, p := range dbPolicyList {
+	if p.ID == "" {
+		if hex := p.MID.Hex(); hex != "" {
+			p.ID = hex
+		} else {
+			p.ID = persistentmodel.NewObjectID().Hex()
+		}
+	}
 	policies[p.ID] = p
 }
Suggestion importance[1-10]: 8

__

Why: Prevents empty-key insertions when parsing RPC data by deriving IDs, restoring robustness previously ensured by MID. Important for data integrity.

Medium
Ensure non-empty policy ID fallback

Guard against cases where newPol.MID is a zero-value and Hex() may return an empty
string, leading to an empty filename and potential overwrite or write errors.
Fallback to generating a stable unique ID (e.g., a new ObjectID) when both ID and
MID.Hex() are empty.

gateway/api.go [936-938]

 if newPol.ID == "" {
-	newPol.ID = newPol.MID.Hex()
+	if hex := newPol.MID.Hex(); hex != "" {
+		newPol.ID = hex
+	} else {
+		newPol.ID = persistentmodel.NewObjectID().Hex()
+	}
 }
Suggestion importance[1-10]: 7

__

Why: Valid enhancement to avoid empty IDs if newPol.MID is zero; improves robustness when creating filenames. Not critical but prevents edge-case failures.

Medium

Copy link

probelabs bot commented Oct 8, 2025
edited
Loading

🔍 Code Analysis Results

This PR standardizes policy identification by deprecating the AllowExplicitPolicyID configuration and consistently using the policy.ID field. It removes the conditional logic that previously switched between policy.ID and the internal MongoDB ID (MID), simplifying policy loading from the Dashboard and RPC. For backward compatibility, a new ensurePolicyId function backfills policy.ID from MID if it's empty.

A major enhancement is the introduction of a new internal/osutil package, which provides sandboxed file system operations. This is used in the policy management API (gateway/api.go) to prevent path traversal vulnerabilities when creating, updating, or deleting policy files. All related function signatures and tests have been updated to reflect these changes.

Files Changed Analysis

  • Total Files Changed: 8
  • Additions/Deletions: 397 additions, 71 deletions.
  • Notable Patterns:
    • New Security Utility: The addition of internal/osutil/osutil.go and its corresponding tests introduces a reusable, secure file I/O module to prevent path traversal.
    • Refactoring for Simplicity: The core logic in gateway/policy.go is simplified by removing the allowExplicit parameter and its associated branching, leading to cleaner code.
    • Security Hardening: gateway/api.go is refactored to replace direct file system calls (ioutil.WriteFile, os.Remove, os.Stat) with the new secure osutil wrapper.
    • API Simplification: Function signatures in gateway/policy.go, gateway/server.go, and gateway/rpc_backup_handlers.go are cleaned up by removing the now-redundant allowExplicit parameter.
    • Test Adaptation: gateway/policy_test.go is significantly modified to align with the new standardized ID handling and simplified function signatures.

Architecture & Impact Assessment

What this PR accomplishes

This PR resolves inconsistencies in how policies are identified, making the gateway's behavior more predictable and the codebase easier to maintain. It eliminates a confusing configuration flag and enforces a single source of truth for policy IDs. Critically, it also hardens the file-based policy management API against path traversal attacks.

Key technical changes introduced

  1. Policy ID Standardization: The system now exclusively uses policy.ID as the unique identifier for policies in memory.
  2. Configuration Deprecation: The policies.allow_explicit_policy_id configuration option is marked as deprecated and is no longer used.
  3. Secure File Operations: A new internal/osutil package sandboxes file system access to a specific root directory, preventing malicious file paths from escaping the intended directory.
  4. Backward Compatibility: The ensurePolicyId function ensures that policies loaded without an ID have it populated from their MID (MongoDB _id), ensuring a smooth transition for existing deployments.
  5. API Simplification: The allowExplicit parameter was removed from LoadPoliciesFromDashboard, LoadPoliciesFromRPC, and parsePoliciesFromRPC, resulting in a cleaner internal API.

Affected system components

  • Gateway Configuration: The allow_explicit_policy_id setting is now obsolete.
  • Policy Loading: Logic for loading policies from the Tyk Dashboard, RPC, and file system is unified.
  • Gateway REST API: The /tyk/policies/* endpoints are now more secure due to sandboxed file I/O.
  • RPC Backup & Restore: The policy restoration mechanism is updated to use the new standardized parsing logic.

Component Interaction Diagrams

Policy ID Handling: Before vs. After

graph TD
 subgraph Before
 A[Policy Source] --> B{allow_explicit_policy_id?}
 B -- "True & ID exists" --> C[Use policy.ID]
 B -- "False or ID empty" --> D[Use policy.MID]
 end
 subgraph After
 E[Policy Source] --> F[ensurePolicyId(policy)]
 F -- "Backfills ID from MID if empty" --> G[Use policy.ID]
 end
Loading

Secure Policy File Handling

sequenceDiagram
 participant API as "Gateway API Handler"
 participant Osutil as "osutil.Root"
 participant FS as "File System"
 API->>Osutil: NewRoot(policyPath)
 API->>Osutil: WriteFile("policy-id.json", data)
 Osutil->>Osutil: Ensure path is within root
 alt Path is safe
 Osutil->>FS: os.WriteFile(fullPath, data)
 else Path is unsafe (e.g., ../../etc/passwd)
 Osutil-->>API: Return error
 API-->>Client: HTTP 500
 end
Loading

Scope Discovery & Context Expansion

The changes are primarily contained within the gateway's policy management subsystem. The most significant architectural change is the introduction of the internal/osutil package. This utility is a valuable addition for security hardening and could be adopted in other parts of the codebase that perform file I/O based on user-provided or API-driven paths (e.g., API definition management).

The deprecation of AllowExplicitPolicyID is a minor breaking change for operators who relied on this niche feature, but the backward-compatibility logic ensures that existing policies will continue to load correctly. The overall impact is a more robust, secure, and maintainable policy management system.

Metadata
  • Review Effort: 3 / 5
  • Primary Label: bug

Powered by Visor from Probelabs

Last updated: 2025年10月14日T13:07:49.473Z | Triggered by: synchronize | Commit: 8ab4b09

Copy link

probelabs bot commented Oct 8, 2025
edited
Loading

🔍 Code Analysis Results

✅ Security Check Passed

No security issues found – changes LGTM.

Performance Issues (4)

Severity Location Issue
🟡 Warning gateway/api.go:926
A new `osutil.Root` object is created on every API request that handles policy files. This involves resolving the absolute path and executing an `os.Stat` call, which introduces unnecessary file system I/O overhead for each request.
💡 SuggestionTo optimize this, the `osutil.Root` object for the policy path should be created once during gateway initialization and cached as a field on the `Gateway` struct. The API handlers can then use this cached object, avoiding the overhead of resolving the path and calling `os.Stat` on every policy management request. The object would only need to be re-initialized if the policy path configuration is changed via a hot reload.
🟡 Warning config/config.go:120
The deprecation comment does not follow the standard Go convention. It should be in the format `// Deprecated: [reason for deprecation].`
💡 SuggestionTo align with Go's best practices for documenting deprecated code, which is also recognized by various tools, please update the comment.
🔧 Suggested Fix
// Deprecated: This configuration is no longer used. The gateway now consistently uses the policy 'ID' field.
🟡 Warning gateway/api.go:947
There is a minor typo in the error log message.
💡 SuggestionCorrecting the typo will improve the clarity of logs, which is helpful for debugging.
🔧 Suggested Fix
log.WithError(err).Error("Unable to create policy root.")
🟡 Warning gateway/policy.go:253
The comment for the `ensurePolicyId` function is too brief and lacks context for future maintainers. It doesn't explain why the function exists or what specific migration it's related to.
💡 SuggestionTo improve maintainability, expand the comment to provide clear context about the function's purpose and the conditions under which it can be removed.
🔧 Suggested Fix
// ensurePolicyId provides backward compatibility for policies that do not have an 'ID' field set.
// It populates the 'ID' from the 'MID' (MongoDB `_id`) if the 'ID' is empty.
// This function can be removed in a future major version once a data migration ensures all policies have an explicit 'ID'. See TT-15825.

Quality Issues (4)

Severity Location Issue
🟡 Warning gateway/api.go:921-929
The `newPolicyPathRoot` function is called on every API request for policy management. This function performs file system I/O (`os.Stat`), which introduces unnecessary overhead in frequently called API handlers.
💡 SuggestionTo optimize this, the `osutil.Root` object for the policy path should be created once during gateway initialization and cached as a field on the `Gateway` struct. The API handlers can then use this cached object, avoiding the overhead of resolving the path and calling `os.Stat` on every policy management request.
🟡 Warning gateway/api.go:947
There is a minor typo in the error log message. It should be "create" instead of "crate".
💡 SuggestionCorrecting the typo will improve the clarity of logs, which is helpful for debugging.
🔧 Suggested Fix
log.WithError(err).Error("Unable to create policy root.")
🟡 Warning config/config.go:120
The deprecation comment does not follow the standard Go convention. It should be in the format `// Deprecated: [reason for deprecation].`
💡 SuggestionTo align with Go's best practices for documenting deprecated code, which is also recognized by various tools, please update the comment.
🔧 Suggested Fix
// Deprecated: This configuration is no longer used. The gateway now consistently uses the policy 'ID' field.
🟡 Warning gateway/policy.go:252-253
The comment for the `ensurePolicyId` function is too brief and lacks context for future maintainers. It doesn't explain why the function exists or what specific migration it's related to.
💡 SuggestionTo improve maintainability, expand the comment to provide clear context about the function's purpose and the conditions under which it can be removed.
🔧 Suggested Fix
// ensurePolicyId provides backward compatibility for policies that do not have an 'ID' field set.
// It populates the 'ID' from the 'MID' (MongoDB `_id`) if the 'ID' is empty.
// This function can be removed in a future major version once a data migration ensures all policies have an explicit 'ID'. See TT-15825.

Style Issues (3)

Severity Location Issue
🟡 Warning unknown:120
The deprecation comment does not follow the standard Go convention. The standard format is `// Deprecated: [reason for deprecation].` which is recognized by static analysis tools and IDEs.
🟡 Warning unknown:947
There is a minor typo in the error log message: "crate" should be "create".
🟡 Warning unknown:253
The comment for the `ensurePolicyId` function is too brief and lacks context for future maintainers. It doesn't explain why the function exists or what specific migration it's related to.

✅ Dependency Check Passed

No dependency issues found – changes LGTM.

✅ Connectivity Check Passed

No connectivity issues found – changes LGTM.


Powered by Visor from Probelabs

Last updated: 2025年10月14日T13:07:50.637Z | Triggered by: synchronize | Commit: 8ab4b09

@shults shults force-pushed the TT-15825-be-address-inconsistencies-with-use-of-policy-identifiers branch 2 times, most recently from 6133cde to fd3b50c Compare October 9, 2025 13:54
@shults shults force-pushed the TT-15825-be-address-inconsistencies-with-use-of-policy-identifiers branch from 6eef7e5 to 5fc2a48 Compare October 14, 2025 12:07
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
73.5% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

AltStyle によって変換されたページ (->オリジナル) /