-
Notifications
You must be signed in to change notification settings - Fork 1.1k
[TT-15825] [BE] Address inconsistencies with use of Policy identifiers #7424
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TT-15825] [BE] Address inconsistencies with use of Policy identifiers #7424
Conversation
PR Reviewer Guide 🔍
Here are some key observations to aid the review process:
Possible Issue
When creating or updating a policy, if the incoming policy has an empty ID
, the code sets newPol.ID
to newPol.MID.Hex()
. Ensure MID
is always populated for new policies; otherwise MID
may be zero-value and .Hex()
could yield an empty or invalid ID, leading to filename collisions or unexpected behavior.
ID
, the code sets newPol.ID
to newPol.MID.Hex()
. Ensure MID
is always populated for new policies; otherwise MID
may be zero-value and .Hex()
could yield an empty or invalid ID, leading to filename collisions or unexpected behavior.if newPol.ID == "" { newPol.ID = newPol.MID.Hex() }
Duplicate Handling
Policies fetched from the Dashboard now strictly use p.ID
as the key and skip duplicates. Validate that upstream always sets ID
(and uniquely) after deprecating explicit/Mongo ID switching; otherwise blank or reused IDs will cause policies to be dropped silently.
p.ID
as the key and skip duplicates. Validate that upstream always sets ID
(and uniquely) after deprecating explicit/Mongo ID switching; otherwise blank or reused IDs will cause policies to be dropped silently.if _, ok := policies[p.ID]; ok { log.WithFields(logrus.Fields{ "prefix": "policy", "policyID": p.ID, "OrgID": p.OrgID, }).Warning("--> Skipping policy, new item has a duplicate ID!") continue } policies[p.ID] = p.ToRegularPolicy() }
Crypto API Change
crypto.Decrypt
is now called with secret
(string/bytes change). Confirm the function signature expects the provided type; mismatches could cause decryption failure and policy-loading errors in backup recovery.
crypto.Decrypt
is now called with secret
(string/bytes change). Confirm the function signature expects the provided type; mismatches could cause decryption failure and policy-loading errors in backup recovery.secret := crypto.GetPaddedString(gw.GetConfig().Secret) cryptoText, err := store.GetKey(checkKey) listAsString := crypto.Decrypt(secret, cryptoText) if err != nil { return nil, errors.New("[RPC] --> Failed to get node policy backup (" + checkKey + "): " + err.Error()) } if policies, err := parsePoliciesFromRPC(listAsString); err != nil { log.WithFields(logrus.Fields{
API Changes
--- prev.txt 2025年10月14日 12:44:46.085105890 +0000 +++ current.txt 2025年10月14日 12:44:36.618225843 +0000 @@ -6879,6 +6879,8 @@ // If you set this value to `true`, then the id parameter in a stored policy (or imported policy using the Dashboard API), will be used instead of the internal ID. // // This option should only be used when moving an installation to a new database. + // + // Deprecated. Is not used in codebase. AllowExplicitPolicyID bool `json:"allow_explicit_policy_id"` // This option only applies in OSS deployment when the `policies.policy_source` is either set // to `file` or an empty string. If `policies.policy_path` is set, then Tyk will load policies @@ -10009,11 +10011,11 @@ func (gw *Gateway) LoadDefinitionsFromRPCBackup() ([]*APISpec, error) -func (gw *Gateway) LoadPoliciesFromDashboard(endpoint, secret string, allowExplicit bool) (map[string]user.Policy, error) +func (gw *Gateway) LoadPoliciesFromDashboard(endpoint, secret string) (map[string]user.Policy, error) LoadPoliciesFromDashboard will connect and download Policies from a Tyk Dashboard instance. -func (gw *Gateway) LoadPoliciesFromRPC(store RPCDataLoader, orgId string, allowExplicit bool) (map[string]user.Policy, error) +func (gw *Gateway) LoadPoliciesFromRPC(store RPCDataLoader, orgId string) (map[string]user.Policy, error) func (gw *Gateway) LoadPoliciesFromRPCBackup() (map[string]user.Policy, error)
PR Code Suggestions ✨Explore these optional code suggestions:
|
🔍 Code Analysis Results
This PR standardizes policy identification by deprecating the AllowExplicitPolicyID
configuration and consistently using the policy.ID
field. It removes the conditional logic that previously switched between policy.ID
and the internal MongoDB ID (MID
), simplifying policy loading from the Dashboard and RPC. For backward compatibility, a new ensurePolicyId
function backfills policy.ID
from MID
if it's empty.
A major enhancement is the introduction of a new internal/osutil
package, which provides sandboxed file system operations. This is used in the policy management API (gateway/api.go
) to prevent path traversal vulnerabilities when creating, updating, or deleting policy files. All related function signatures and tests have been updated to reflect these changes.
Files Changed Analysis
- Total Files Changed: 8
- Additions/Deletions: 397 additions, 71 deletions.
- Notable Patterns:
- New Security Utility: The addition of
internal/osutil/osutil.go
and its corresponding tests introduces a reusable, secure file I/O module to prevent path traversal. - Refactoring for Simplicity: The core logic in
gateway/policy.go
is simplified by removing theallowExplicit
parameter and its associated branching, leading to cleaner code. - Security Hardening:
gateway/api.go
is refactored to replace direct file system calls (ioutil.WriteFile
,os.Remove
,os.Stat
) with the new secureosutil
wrapper. - API Simplification: Function signatures in
gateway/policy.go
,gateway/server.go
, andgateway/rpc_backup_handlers.go
are cleaned up by removing the now-redundantallowExplicit
parameter. - Test Adaptation:
gateway/policy_test.go
is significantly modified to align with the new standardized ID handling and simplified function signatures.
- New Security Utility: The addition of
Architecture & Impact Assessment
What this PR accomplishes
This PR resolves inconsistencies in how policies are identified, making the gateway's behavior more predictable and the codebase easier to maintain. It eliminates a confusing configuration flag and enforces a single source of truth for policy IDs. Critically, it also hardens the file-based policy management API against path traversal attacks.
Key technical changes introduced
- Policy ID Standardization: The system now exclusively uses
policy.ID
as the unique identifier for policies in memory. - Configuration Deprecation: The
policies.allow_explicit_policy_id
configuration option is marked as deprecated and is no longer used. - Secure File Operations: A new
internal/osutil
package sandboxes file system access to a specific root directory, preventing malicious file paths from escaping the intended directory. - Backward Compatibility: The
ensurePolicyId
function ensures that policies loaded without anID
have it populated from theirMID
(MongoDB_id
), ensuring a smooth transition for existing deployments. - API Simplification: The
allowExplicit
parameter was removed fromLoadPoliciesFromDashboard
,LoadPoliciesFromRPC
, andparsePoliciesFromRPC
, resulting in a cleaner internal API.
Affected system components
- Gateway Configuration: The
allow_explicit_policy_id
setting is now obsolete. - Policy Loading: Logic for loading policies from the Tyk Dashboard, RPC, and file system is unified.
- Gateway REST API: The
/tyk/policies/*
endpoints are now more secure due to sandboxed file I/O. - RPC Backup & Restore: The policy restoration mechanism is updated to use the new standardized parsing logic.
Component Interaction Diagrams
Policy ID Handling: Before vs. After
graph TD
subgraph Before
A[Policy Source] --> B{allow_explicit_policy_id?}
B -- "True & ID exists" --> C[Use policy.ID]
B -- "False or ID empty" --> D[Use policy.MID]
end
subgraph After
E[Policy Source] --> F[ensurePolicyId(policy)]
F -- "Backfills ID from MID if empty" --> G[Use policy.ID]
end
Secure Policy File Handling
sequenceDiagram
participant API as "Gateway API Handler"
participant Osutil as "osutil.Root"
participant FS as "File System"
API->>Osutil: NewRoot(policyPath)
API->>Osutil: WriteFile("policy-id.json", data)
Osutil->>Osutil: Ensure path is within root
alt Path is safe
Osutil->>FS: os.WriteFile(fullPath, data)
else Path is unsafe (e.g., ../../etc/passwd)
Osutil-->>API: Return error
API-->>Client: HTTP 500
end
Scope Discovery & Context Expansion
The changes are primarily contained within the gateway's policy management subsystem. The most significant architectural change is the introduction of the internal/osutil
package. This utility is a valuable addition for security hardening and could be adopted in other parts of the codebase that perform file I/O based on user-provided or API-driven paths (e.g., API definition management).
The deprecation of AllowExplicitPolicyID
is a minor breaking change for operators who relied on this niche feature, but the backward-compatibility logic ensures that existing policies will continue to load correctly. The overall impact is a more robust, secure, and maintainable policy management system.
Metadata
- Review Effort: 3 / 5
- Primary Label: bug
Powered by Visor from Probelabs
Last updated: 2025年10月14日T13:07:49.473Z | Triggered by: synchronize | Commit: 8ab4b09
🔍 Code Analysis Results✅ Security Check PassedNo security issues found – changes LGTM. Performance Issues (4)
Quality Issues (4)
Style Issues (3)
✅ Dependency Check PassedNo dependency issues found – changes LGTM. ✅ Connectivity Check PassedNo connectivity issues found – changes LGTM. Powered by Visor from Probelabs Last updated: 2025年10月14日T13:07:50.637Z | Triggered by: synchronize | Commit: 8ab4b09 |
6133cde
to
fd3b50c
Compare
6eef7e5
to
5fc2a48
Compare
Quality Gate Failed Quality Gate failed
Failed conditions
73.5% Coverage on New Code (required ≥ 80%)
Uh oh!
There was an error while loading. Please reload this page.
User description
TT-15825
Description
Related Issue
Motivation and Context
How This Has Been Tested
Screenshots (if appropriate)
Types of changes
Checklist
PR Type
Bug fix, Tests, Documentation
Description
Remove explicit policy ID toggle usage
Standardize policy ID to
policy.ID
Update RPC/Dashboard loaders signatures
Adjust tests for new behavior
Diagram Walkthrough
File Walkthrough
config.go
Deprecate unused AllowExplicitPolicyID config
config/config.go
AllowExplicitPolicyID
as deprecated.api.go
Backfill missing policy ID from MID on save
gateway/api.go
newPol.ID
tonewPol.MID.Hex()
if empty.policy.go
Standardize ID handling and simplify loader APIs
gateway/policy.go
allowExplicit
parameter from loaders/parsers.policy.ID
as key; drop MID fallback.policy.ID
.rpc_backup_handlers.go
Update RPC backup to new parsing/decrypt API
gateway/rpc_backup_handlers.go
parsePoliciesFromRPC
without allowExplicit.server.go
Remove allowExplicit parameter from sync paths
gateway/server.go
AllowExplicitPolicyID
.policy_test.go
Align tests with standardized policy ID handling
gateway/policy_test.go
policy.ID
.AllowExplicitPolicyID
.