Optionally keep paths that do not exist with resolve #11

UserCurt commented 2025年10月11日 11:41:42 +02:00

This feature request is to optionally allow mpath.resolve to return MPath Pointers that do not point to existing entries (i.e., the paths do not exist).

The proposed solution is to replace the named argument AllowEmptyFields with a new one NullField which dictates behavior when a null field is encountered. When a field that does not exist is encountered, the behavior will be dictated by the value of Nullfield:

• omit -- Ignores the invalid/null path and excludes it from the result
• missing -- Replaces the invalid/null path with a missing Pointer
• retain -- Keeps the invalid/null path
• error -- Gives an error

There is some overlap in functionality between this proposal and FillGaps with some nuanced differences when working with heterogeneous data structures. These details may not matter to the end user, so merging FillGaps with NullField should be considered.

This feature request is to optionally allow `mpath.resolve` to return MPath Pointers that do not point to existing entries (i.e., the paths do not exist). The proposed solution is to replace the named argument `AllowEmptyFields` with a new one `NullField` which dictates behavior when a null field is encountered. When a field that does not exist is encountered, the behavior will be dictated by the value of `Nullfield`: - `omit` -- Ignores the invalid/null path and excludes it from the result - `missing` -- Replaces the invalid/null path with a missing Pointer - `retain` -- Keeps the invalid/null path - `error` -- Gives an error There is some overlap in functionality between this proposal and `FillGaps` with some nuanced differences when working with heterogeneous data structures. These details may not matter to the end user, so merging `FillGaps` with `NullField` should be considered.

UserCurt commented 2025年10月11日 11:54:42 +02:00

FillGaps might need to be generalized to include missing fields:

S = struct(A=struct(X=1, Y=2), B=struct(Y=3, Z=4));
mpath.resolve(S, "/*/*")
% returns:
% [/A/X, /A/Y, <missing>;
% <missing>, /B/Y, /B/Z ]

The open question is what option (named argument) should be provided so that the user can get this instead:

% [/A/X, /A/Y, /A/Z;
% /B/X, /B/Y, /B/Z ]

A related question: Should /{A,B}/{X,Y,Z} (once set notation from #3 is implemented) behave identically to /*/* in this circumstance? I can see the argument how the fields X, Y, and Z must be checked for each branch A and B in the former, but for the later each * at each branch is considered independent and can match different fields. However, everything eventually has to be shaped anyways, so the second * effectively share the same pool of fields, suggesting these should behave the same. This line of reasoning suggests FillGaps and NullField should be merged somehow. That is, unless there is a use case otherwise?

`FillGaps` might need to be generalized to include missing fields: ```matlab S = struct(A=struct(X=1, Y=2), B=struct(Y=3, Z=4)); mpath.resolve(S, "/*/*") % returns: % [/A/X, /A/Y, <missing>; % <missing>, /B/Y, /B/Z ] ``` The open question is what option (named argument) should be provided so that the user can get this instead: ```matlab % [/A/X, /A/Y, /A/Z; % /B/X, /B/Y, /B/Z ] ``` A related question: Should `/{A,B}/{X,Y,Z}` (once set notation from #3 is implemented) behave identically to `/*/*` in this circumstance? I can see the argument how the fields X, Y, and Z must be checked for each branch A and B in the former, but for the later each * at each branch is considered independent and can match different fields. However, everything eventually has to be shaped anyways, so the second `*` effectively share the same pool of fields, suggesting these should behave the same. This line of reasoning suggests `FillGaps` and `NullField` should be merged somehow. That is, unless there is a use case otherwise?

UserCurt commented 2026年06月25日 04:03:14 +02:00

Some progress has been made on this issue but it is not yet complete.

Two notes:

• Having resolve be able to return paths that do not exist will help enable set to create fields that do not exist.
• FillGaps was historically overloaded in get to handle two distinct situations: Filling in gaps where paths were missing (via missing paths returned from resolve) and filling in the gaps when there was a size mismatch. It was later split into the now MissingPolicy and SizeMismatchPolicy.

Relevant history around resolve:

• 137593e on 2025年09月25日: AllowEmptyMatches and FillGaps originally introduced using logical policy.
• 5df9c181 on 2025年10月10日: This was changed where AllowEmptyMatches was replaced with NullField in resolve with more options, now supporting for "omit" "missing", "retain", "error".
• 70f5380 on 2025年10月16日: Prior to this point, NullField determined what to do when a missing path branch was found, while FillGaps determined whether post-reshape rectangular holes were allowed. In this commit, NullField was renamed (and old AllowEmptyMatches language) to MissingPolicy in resolve, and FillGaps was removed from resolve.

The new MissingPolicy wasn't fully functional at this time but was addressed in later commits:

• bd5ddcd: retained path reconstruction for missing resolved slots
• df3e54f: heterogeneous struct-array missing-policy bug
• 65af1e0: missing branch handling
• 84f9e78: DotSet with MissingPolicy="retain" including missing fields

The new MissingPolicy approach was then used with get and other functions:

• 90a851f on 2026年06月11日: While some changes were being underwent with resolve, get was disabled. This commit re-enabled get.
• 70ff089 on 2026年06月13日: Policy semantics in get became better controlled by having MissingPolicy for handling missing paths and SizeMismatchPolicy for handling mismatched sizes in the results.
• 206e68a and 83e1046 on 2026年06月13日: set and remove were modified to use MissingPolicy but with different options based on what was reasonable for those functions: set accepted "create", "ignore", and "error", and remove accepted "ignore" and "error".

---

This issue is not fully addressed yet. Below is an example that does not return results as expected:

>> data = struct(Y = struct(A = 1, B = 2));
>> mp = mpath.Pointer([mpath.operation.DotSet(["X", "Y"]) mpath.operation.DotGlob("*")])
mp = 
 mpath: /{X,Y}/*
>> mp.resolve(data, MissingPolicy='omit')
ans = 
 1×ばつ2 mpath array
 /Y/A /Y/B
>> mp.resolve(data, MissingPolicy='missing')
ans = 
 2×ばつ3 mpath array
 <missing> <missing> <missing>
 <missing> /Y/A /Y/B
>> mp.resolve(data, MissingPolicy='retain')
ans = 
 2×ばつ3 mpath array
 /X/* /X/A /X/B
 /Y/* /Y/A /Y/B

The expected results would be:

>> mp.resolve(data, MissingPolicy='omit')
ans = 
 ×ばつ2 mpath array
 /Y/A /Y/B
>> mp.resolve(data, MissingPolicy='missing')
ans = 
 ×ばつ3 mpath array
 <missing> <missing>
 /Y/A /Y/B
>> mp.resolve(data, MissingPolicy='retain')
ans = 
 ×ばつ3 mpath array
 /X/A /X/B
 /Y/A /Y/B

Some progress has been made on this issue but it is not yet complete. Two notes: - Having `resolve` be able to return paths that do not exist will help enable `set` to create fields that do not exist. - `FillGaps` was historically overloaded in `get` to handle two distinct situations: Filling in gaps where paths were missing (via missing paths returned from `resolve`) and filling in the gaps when there was a size mismatch. It was later split into the now `MissingPolicy` and `SizeMismatchPolicy`. Relevant history around `resolve`: - `137593e` on `2025年09月25日`: `AllowEmptyMatches` and `FillGaps` originally introduced using logical policy. - `5df9c181` on `2025年10月10日`: This was changed where `AllowEmptyMatches` was replaced with `NullField` in `resolve` with more options, now supporting for `"omit"` `"missing"`, `"retain"`, `"error"`. - `70f5380` on `2025年10月16日`: Prior to this point, `NullField` determined what to do when a missing path branch was found, while `FillGaps` determined whether post-reshape rectangular holes were allowed. In this commit, `NullField` was renamed (and old `AllowEmptyMatches` language) to `MissingPolicy` in `resolve`, and `FillGaps` was removed from `resolve`. The new `MissingPolicy` wasn't fully functional at this time but was addressed in later commits: - `bd5ddcd`: retained path reconstruction for missing resolved slots - `df3e54f`: heterogeneous struct-array missing-policy bug - `65af1e0`: missing branch handling - `84f9e78`: `DotSet` with `MissingPolicy="retain"` including missing fields The new `MissingPolicy` approach was then used with `get` and other functions: - `90a851f` on `2026年06月11日`: While some changes were being underwent with `resolve`, `get` was disabled. This commit re-enabled `get`. - `70ff089` on `2026年06月13日`: Policy semantics in `get` became better controlled by having `MissingPolicy` for handling missing paths and `SizeMismatchPolicy` for handling mismatched sizes in the results. - `206e68a` and `83e1046` on `2026年06月13日`: `set` and `remove` were modified to use `MissingPolicy` but with different options based on what was reasonable for those functions: `set` accepted `"create"`, `"ignore"`, and `"error"`, and `remove` accepted `"ignore"` and `"error"`. --- This issue is not fully addressed yet. Below is an example that does not return results as expected: ```matlab >> data = struct(Y = struct(A = 1, B = 2)); >> mp = mpath.Pointer([mpath.operation.DotSet(["X", "Y"]) mpath.operation.DotGlob("*")]) mp = mpath: /{X,Y}/* >> mp.resolve(data, MissingPolicy='omit') ans = ×ばつ2 mpath array /Y/A /Y/B >> mp.resolve(data, MissingPolicy='missing') ans = ×ばつ3 mpath array <missing> <missing> <missing> <missing> /Y/A /Y/B >> mp.resolve(data, MissingPolicy='retain') ans = ×ばつ3 mpath array /X/* /X/A /X/B /Y/* /Y/A /Y/B ``` The expected results would be: ``` >> mp.resolve(data, MissingPolicy='omit') ans = ×ばつ2 mpath array /Y/A /Y/B >> mp.resolve(data, MissingPolicy='missing') ans = ×ばつ3 mpath array <missing> <missing> /Y/A /Y/B >> mp.resolve(data, MissingPolicy='retain') ans = ×ばつ3 mpath array /X/A /X/B /Y/A /Y/B ```

UserCurt commented 2026年07月02日 11:07:05 +02:00

The issue has been addressed in fdd36bb.

Current behavior from the example in the comment above gives the expected result shapes:

• MissingPolicy="omit" -> ×ばつ2
• MissingPolicy="missing" -> ×ばつ2
• MissingPolicy="retain" -> ×ばつ2

Related but more specific bugs are being tracked separately:

• #32 for paths that continue through non-container values
• #31 for behavior with empty resolved arrays

The issue has been addressed in `fdd36bb`. Current behavior from the example [in the comment above](https://codeberg.org/UserCurt/mpath/issues/11#issuecomment-18031568) gives the expected result shapes: - `MissingPolicy="omit"` -> ×ばつ2` - `MissingPolicy="missing"` -> ×ばつ2` - `MissingPolicy="retain"` -> ×ばつ2` Related but more specific bugs are being tracked separately: - #32 for paths that continue through non-container values - #31 for behavior with empty resolved arrays