homepage

Following on from Guido's rejection of PEP 410:
http://mail.python.org/pipermail/python-dev/2012-February/116837.html
This bug is the proposed hammering-out space for how to preserve exact metadata for st_atime / st_mtime between os.stat and os.utime.
(Yes, there's already #11457 -- but that went pretty far down the rabbit hole of Decimal. I thought maybe a fresh start would be best.)

Guido proposed st_atime_ns et al. I'll make an alternate proposal.
I'd like to avoid tying ourselves to ns resolution. As MvL said: "I don't want to deal with this issue *again* in my lifetime".
If all we want is to facilitate copying the exact metadata from os.stat to os.utime, then we don't really care about conveniently modifying the timestamp. So I propose we use MvL's suggestion of a tuple of ints. Either (numerator, denominator) or (seconds, fractional_numerator, fractional_denominator). (These are mentioned in PEP 410 under the heading "Tuple of ints" as options A and B respectively.) Name the fields with the suffix "_exact" (e.g. st_mtime_exact). os.stat and its ilk produce it; os.utime and its ilk consume it. If people want to monkey with the values and do math, let 'em--consenting adults.

OTOH, it seems that Guido is very much in favor of hard-coding ns resolution in the API, claiming that expectations of even finer resolution are academic. While I still disagree (the *very* same argument was brought up for ms resolution ten years ago), I think that practicality beats purity here: people apparently can deal with an fixed scale much better than with a floating one. So it may be better to meet the apparent intuition of the majority than follow the purity route.

Is it totally insane to think about using a float subclass with an additional attribute instead of a tuple?

@RDM: yes, that's insane for this purpose. The size of the solution doesn't match the size of the problem (the problem is non-existent for most people).
@Larry: That seems unnecessarily general. I take full responsibility for fixing the precision at ns. And note that I *am* giving you a way out -- I'm establishing a clear pattern, and if someone really, desperately wants ps or fs, it's completely obvious how to change the API again. But I bet you a case of beer that won't happen in our lifetimes (or in Benjamin's). So the dynamic precision is just unnneeded complexity.

I suggest that publishing nanoseconds as a plain int would be a nasty API. Consider what it would do to os.utime:
 if isinstance(mtime, int):
 # must be st_mtime_ns, it's in nanoseconds, use as-is
 value = mtime
 else:
 assert isinstance(mtime, float)
 # must be st_mtime, it's in seconds, multiply by a billion
 value = mtime * 1000000000
Have we ever published an API that treated a parameter as two wildly different numbers based solely on whether the parameter was an int or a float?

(D'oh. My pseudo-code should have said
 value = int(mtime * 1000000000)
for that second assignment. Hopefully my point was still clear.)

How about a separate float attribute for the fractional part (in seconds)? This gives a femtosecond precision (assuming a 50 bits mantissa). The utime() could accept a (integral part, fractional part) tuple to set a timestamp without losing bits.
(PHP does something similar, but badly, with its microtime() function:
http://www.php.net/manual/en/function.microtime.php)

@pitrou: What would that look like when passed in to os.utime?

>>> st = os.stat('LICENSE')
>>> st.st_mtime
1330108216.7984242
>>> st.st_mtime_frac
0.798424152
>>> tup = st.st_mtime, st.st_mtime_frac
>>> os.utime('LICENSE', (tup, tup))
Of course, the fact that utime takes a (atime, mtime) tuple makes this a bit cumbersome.
When passed a tuple, utime would ignore the fractional part of the first element.

> I suggest that publishing nanoseconds as a plain int would be a 
> nasty API. Consider what it would do to os.utime:
No, it wouldn't. Please re-read Guido's proposal. If you want to
specify nanoseconds, you have to pass the ns= parameter. My only
quibble with the specific spelling is that it invokes Godwin's law
(but I can live that that as a theoretical concern, also).
> Have we ever published an API that treated a parameter as two wildly 
> different numbers based solely on whether the parameter was an int 
> or a float?
No, and Guido is on the record for objecting such APIs. Hence the
keyword parameter.

Thanks Martin. I'd be happy with nsec instead of ns.

My mistake! That ought to teach me to bound out of bed Sunday morning with my "brilliant realization". (But it probably won't.)
I volunteer to implement this, with the new custom class for the os.stat object. I'll have it done no later than the PyCon sprints next month.
For the record, I actually prefer "ns". "s" is the SI standard abbreviation for second (as is "n" for nano-):
http://en.wikipedia.org/wiki/International_System_of_Units#Units_and_prefixes
If we're going to abbreviate, I think it's reasonable to go with their standard. But I'm willing to be overridden.

Let's go with ns= them. It's also what POSIX uses (although I initially had problems parsing "futimens", reading it as foo-timen-z, when it really is eff-youtime-en-es)

Because the use case is to copy the access and modification time from a file to another, I would prefer to use the timespec structure: (sec: int, nsec: int). st_atime_ns and st_mtime_ns fields would be added to os.stat() structure: int in range [0; 999999999].
Copy atime and mtime from src to dst:
st = os.stat(src)
atime = (math.floor(st.st_atime), st.st_atime_ns)
mtime = (math.floor(st.st_mtime), st.st_mtime_ns)
os.utime(dst, (atime, mtime))
os.utime() would accept int, float or (sec: int, nsec: int) for atime and mtime. Examples:
os.utime(name, (1, 1))
os.utime(name, (1.0, 1.0))
os.utime(name, ((1, 0), (1, 0)))

> os.utime() would accept int, float or (sec: int, nsec: int) for atime and mtime.
That's not future-proof for when we have better-than-nanosecond
timestamps. See my suggestion above.

Actually, I'm hoping that by the time we get better than nanosecond resolution, we can also switch to 128-bit floats, and then the existing st_[acm]time field will become the preferred representation once more. What goes around comes around!

> Actually, I'm hoping that by the time we get better than nanosecond
> resolution, we can also switch to 128-bit floats, and then the
> existing st_[acm]time field will become the preferred representation
> once more.
What if your hope isn't fulfilled? That doesn't sound like a reasonable
decision-making strategy.

Well, Guido has already nixed future-proof formats, see his comments above:
"I take full responsibility for fixing the precision at ns."
So hope is all I have left.

> Well, Guido has already nixed future-proof formats, see his comments
> above:
I don't think Guido is *against* future-proof formats per se, he's
against them when they have a cost compared to non future-proof ones.
The proposal I made (a (integral part, float fractional part) tuple)
doesn't have a cost compared to the plain (int seconds, int nanoseconds)
tuple proposal.
(of course, you can also have nanoseconds as a float, but that starts
being weird)

I grant you that (int, float) is probably, theoretically better than (int, int). But it's academic as Guido has ruled against anything but a straight int representing nanoseconds, and I doubt he's gonna change his mind.

Any solution involving tuple is too ugly. Please stick with the plan.

> I grant you that (int, float) is probably, theoretically better than
> (int, int). But it's academic as Guido has ruled against anything but
> a straight int representing nanoseconds, and I doubt he's gonna change
> his mind.
Why not let Guido speak out?

The following solution might be compatible with Guido's suggestion:
os.stat(path).st_atime_ns -> nanoseconds_since_epoch_as_int
os.stat(path).st_ctime_ns -> nanoseconds_since_epoch_as_int
os.stat(path).st_mtime_ns -> nanoseconds_since_epoch_as_int
atime = os.stat(path).st_atime_ns
mtime = os.stat(path).st_mtime_ns
os.utime(path, (atime, mtime), resolution="ns")

I don't see how that's better than
os.utime(path, ns=(atime, mtime))
If you think that in the future you could add resolution="fs", well, you could just as easily add fs=(atime, mtime).

FWIW, +1 on using the *name* of the keyword arg to define the format/resolution of the argument. It's simple and clear right now, and is easily updated to handle higher resolutions in the future.

I'm lost in all issues related to os.stat/utime and nanosecond, here is a list:
 - #10148: duplicate
 - #11457: closed (was related the the rejected PEP 410)
 - #12904: closed, it was the first step to fix os.stat/os.utime
 - #13882: closed (implementation of the PEP 410)
 - #13964: duplicate
 - #14127: this issue, open :-)

The following changes has to be done to fix this issue:
 - add st_atime_ns, st_mtime_ns and st_ctime_ns fileds to os.stat() result: number of nanoseconds since Epoch (1970年1月1日), an integer
 - change os.*utime*() functions (see below)
 - shutil.copystat() should use os.utime(ns=...) and os.ltime(ns=...)
List of the os.*utime*() functions:
 - os.futimes(): use futimens() or futimes()
 - os.futimens(): use futimens(); UTIME_NOW and UTIME_OMIT flags
 - os.futimesat(): use utimensat() or futimesat()
 - os.lutimes(): use futimesat(AT_SYMLINK_NOFOLLOW) or lutimes()
 - os.utime(): use SetFileTime() (Windows), utimensat(), utimes() or utime()
 - os.utimensat(): use utimensat(); UTIME_NOW and UTIME_OMIT flags
Changes on the os.*utime*() functions:
 - add ns keyword to:
 * os.futimes()
 * os.futimesat()
 * os.lutimes()
 * os.utime()
 - except a number of nanoseconds instead of a number of seconds:
 * os.futimens()
 * os.utimensat()
The ns keyword is an exclusive parameter with existing times parameter. Examples:
 * seconds: os.utime(name, (1, 2))
 * seconds: os.utime(name, times=(1, 2))
 * nanoseconds: os.utime(name, ns=(1, 2))
 * INVALID! os.utime(name, (1, 2), ns=(1, 2))
 * INVALID! os.utime(name, times=(1, 2), ns=(1, 2))
I don't want to remove os.futimens() and os.utimensat() because they add a feature: UTIME_NOW and UTIME_OMIT flags.

Larry, are you sprinting on this? I'd love to help.

> I don't want to remove os.futimens() and os.utimensat() because they add a feature: UTIME_NOW and UTIME_OMIT flags.
I'm not sure how this could work: UTIME_NOW and UTIME_OMIT have 
typically values such as ((1l << 30) - 2l) which could be mistaken
as a time stamp if there is a flat nanosecond value.
There would be ways to solve this, of course: not passing the
value should be considered as UTIME_OMIT, and passing -1 may
be treated as UTIME_NOW.

> Larry, are you sprinting on this? I'd love to help.
I am. Come on by--mi CPU es tu CPU!

Guido suggested I break the work up into small patches. So here's the first patch: adds st_?time_ns fields to os.stat() result. Includes doc changes and a reasonable smoke-test in the regrtest suite; suite passes all expected tests. FYI I wasn't sure what to call the currently-slightly-uncomfortably-named new function _PyTime_LongFromTime_t().

Posted a review on Rietveld.

> FYI I wasn't sure what to call the currently-slightly-uncomfortably-named new function _PyTime_LongFromTime_t().
There is already a private _PyLong_FromTime_t() function in
_testcapi.c: just move the function to pytime.c (but keep the _Py
prefix).

I added a review as well. Not sure if that sends email by itself.

> I added a review as well. Not sure if that sends email by itself.
It's supposed to send e-mail, but sometimes not all recipients are spelt
right...

I was notified by email of both Antoine's and Guido's reviews.

My revised patch, incorporating changes suggested by Antoine and Guido.

Somehow patch #2 doesn't show up in Rietveld. :-(
Anyway, I'll leave completion of the review to Antoine, who's more competent in this area. :-)

Attached is a second patch that adds keyword-only argument support to PyArg_ParseTupleAndKeywords. This was developed in isolation.
The magic character is '$'; I would have used '*', but we already use '*' in the format string as a modifier to 'U' and 's' etc. If someone has a better idea for the format I'd be interested. Martin suggested using '|' a second time but I am unsure.
(Re: no reitveld link: my repo must have drifted too far and the patch didn't align cleanly. I've freshened and am rerunning the regression test; once that runs clean I'll upload a fresh patch.)

> Attached is a second patch that adds keyword-only argument support to
> PyArg_ParseTupleAndKeywords. This was developed in isolation.
Please post that on a dedicated issue :)

Do you insist? It's a small patch, and I need it as a precondition to posting the... *third* patch in this series, which adds the ns= parameter to utime and its brothers. (Most of the patch is test and doc; it really only adds about twenty lines of C.)

A fresh patch for adding st_mtime_ns etc to os.stat output. This is the same as last night's patch but with fresh line numbers. (Generated against 44eba26951f6.)

> Do you insist? It's a small patch, and I need it as a precondition to
> posting the...
Yes, I do. It might a small patch, but it changes a core API and should
be discussed on its own.

Posted a last review on Rietveld.

I've created a separate issue for adding keyword-only parameter support to PyArg_ParseTupleAndKeywords, #14328. I've also posted a new patch to that issue incorporating Greg's suggestions. Please file all further comments regarding this topic on that issue.

Updated patch incorporating only one of Antoine's latest suggestions. Please see my reply on Reitveld for r3 as to why I skipped two of your three suggestions.

Incorporated Greg's (hopefully) parting shot. Also put the "as an integer" back for the docs. Finally, munged the note about float seconds vs integer nanoseconds a bit; I think it's an improvement.
Can I get the go ahead now, pretty please?

Revision 7 of my patch. Incorporates all suggestions from Georg. Added "as an integer" in the docs to make Antoine happy.
If I fail to create the "billion" int in INITFUNC, I return NULL and leak references. I figure this is fine, as a) the other such tests in this function also leak, and b) if you can't initialize the os module, you're not going to have a very productive session.
Dare I dream that it's ready for checkin?

> Revision 7 of my patch.
- assert(sizeof(time_t) <= sizeof(long));
You should keep this assertion in _PyLong_FromTime_t().

Incorporated Victor's reinstated assert suggestion. Fingers crossed.

The patch looks good to me. I was going to commit it but then I realized that Larry is a core developer. Last nits: you should add a newline after } in "} else {", and it would be better to move billion variable into fill_time(). Something like:
static void
fill_time(PyObject *v, int index, time_t sec, unsigned long nsec)
{
 ...
 static PyObject *billion = NULL;
 if (billion == NULL) {
 billion = PyLong_FromLong(1000000000);
 if (!billion)
 return;
 }
 ...
}

> The patch looks good to me. I was going to commit it
> but then I realized that Larry is a core developer.
Indeed; please permit me the pleasure of checking it in myself. That is, assuming I actually get the go-ahead someday. (Probably in the far-flung future. I'm imagining my house CPU telling me the good news via my cranial implant. Perhaps while I'm landing my jet car!)
> Last nits: you should add a newline after } in "} else {",
1/3 of the "else {" statements in the Python tree are "} else {". And that's not a new line with me; I merely renamed the variable.
But I just checked, and PEP 7 says so, and I figure it's harmless enough, so I have made that change.
> and it would be better to move billion variable into fill_time().
That's how it was until patch 5 or 6 or so. Greg P. Smith suggested moving into INITFUNC and I agreed. I think it should stay where it is.
Attached is patch #9, adding a newline between "}" and "else" as Victor suggests.

Sorry to let this slide but I just got back from vacation. Unless anybody has any more feedback I'm checking this in tomorrow (Wednesday April 4). Phew!

New changeset f554043badec by Larry Hastings in branch 'default':
Issue #14127: Add st_{cma}time_ns fields to os.stat() result object.
http://hg.python.org/cpython/rev/f554043badec 

Sorry about the delay; laptop died, finally dealt with reviving the data off it. Also: fixed spelling error in title.

Attached is round 1 of my patch adding the ns= parameter to utime, futimes, and lutimes.
Some notes:
* I admit the "see utime for use of times and ns" documentation dodge
 (for both Doc and docstring) is lazy. Yet I'm still hoping to get
 away with it.
* I removed futimens because it's now extraneous.
* I didn't add ns= to utimensat or futimesat. That's because I'm hoping to get rid of them via #14626. (And lutimes too, come to think of it!) I hereby promise that if they survive to June 15th I'll add ns= support
before 3.3 hits beta.
* At last, shutil.copystat (and therefore shutil.copy2) now *exactly*
 preserves timestamps. :D

> I removed futimens because it's now extraneous.
futimens() has nice feature: it is possible to only update atime only update mtime, or use "now" as the new atime and/or mtime.
	
 3882 If *_nsec is specified as UTIME_NOW, the timestamp is updated to the\n\	
 3883 current time.\n\	
 3884 If *_nsec is specified as UTIME_OMIT, the timestamp is not updated.
os.utimensat() has the same feature. Is it possible that os.utimensat() is not available whereas os.futimens() is availalble?

> futimens() has nice feature: it is possible to only update atime
> only update mtime, or use "now" as the new atime and/or mtime.
YAGNI. Worst case, you can use call futimes twice, once with no args, then fstat() it to get the current-ish time and rewrite the fields selectively.
Do you have code where you selectively use UTIME_NOW for only one of the two fields?

Second round of my patch, incorporating nearly all of Victor's suggestions. Thanks, Victor!

Anybody else? I guess I'm gonna juuuust miss Alpha 3, but if nobody has any other objections I'll check this in today (Thursday).
If you want me to hold off just let me know.

> Somehow patch #2 doesn't show up in Rietveld. :-(
It's because it's a git-style diff, so it includes no base revision,
and it didn't apply cleanly to the default head (which is tried as
a fall-back in case of a missing base revision).

FYI, Martin was replying to Guido's comment from more than a month ago, when I posted revision #2 of my first patch series. By sheer coincidence I posted revision #2 of a new patch series yesterday. But Reitveld worked fine for that!
Anyway--no comments? Normally I find the patch review process akin to getting pecked to death by ducks. It's hard to believe this one might go in after only one revision. Somebody pinch me!

Failure on x86 OpenIndiana 3.x:
FAIL: test_stat_attributes (test.test_os.StatAttributeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_os.py", line 240, in test_stat_attributes
 self.check_stat_attributes(self.fname)
 File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_os.py", line 199, in check_stat_attributes
 self.assertEqual(floaty, nanosecondy)
AssertionError: 133572199178458 != 133572199178457
http://www.python.org/dev/buildbot/all/builders/x86%20OpenIndiana%203.x/builds/3494/steps/test/logs/stdio 

New changeset bba131e48852 by Larry Hastings in branch 'default':
Issue #14127: Add ns= parameter to utime, futimes, and lutimes.
http://hg.python.org/cpython/rev/bba131e48852 

@haypo: Thanks for pointing that out buildbot failure! The OpenIndiana buildbot was bit by a rounding error. I fixed it by adding in a fudge factor it--I snuck in one last change just now. I weakened the unit test as follows:
- self.assertEqual(floaty, nanosecondy)
+ self.assertAlmostEqual(floaty, nanosecondy, delta=2)
Also: I'm leaving this issue open for now, to remind myself to add ns= to utimensat and futimesat if they're still alive on June 15th.

> The OpenIndiana buildbot was bit by a rounding error.
How do we have rounding issue with only integers?

We don't!
The test that failed compares the ns_[amc]time and ns_[amc]time_ns fields to ensure they're roughly equivalent. Note that the former fields are floats, which by now ought not to be news to you.

http://www.python.org/dev/buildbot/all/builders/AMD64%20OpenIndiana%203.x/builds/3388/steps/test/logs/stdio
ERROR: test_copy2_symlinks (test.test_shutil.TestShutil)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_shutil.py",
line 328, in test_copy2_symlinks
 shutil.copy2(src_link, dst, symlinks=True)
 File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/shutil.py",
line 193, in copy2
 copystat(src, dst, symlinks=symlinks)
 File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/shutil.py",
line 157, in copystat
 utime_func(dst, ns=(st.st_atime_ns, st.st_mtime_ns))
TypeError: _nop() got an unexpected keyword argument 'ns'

New changeset cdc4e0f8135d by Larry Hastings in branch 'default':
Issue #14127: Fix no-op stub for platforms that lack some "os" functions.
http://hg.python.org/cpython/rev/cdc4e0f8135d 

bba131e48852 causes crashes on Windows.
The attached patch fixes the crash and makes test_os pass for me.
However, using "PyErr_ExceptionMatches(PyExc_RuntimeError)" to check whether to try again using narrow strings is ugly. Maybe utime_read_time_arguments() should be changed to have three possible return values.

> bba131e48852 causes crashes on Windows.
>
> The attached patch fixes the crash and makes test_os pass for me.
>
> However, using "PyErr_ExceptionMatches(PyExc_RuntimeError)" to check
> whether to try again using narrow strings is ugly. Maybe
> utime_read_time_arguments() should be changed to have three possible
> return values.
I appreciate the feedback, and the patch. And I agree--we should be able to find a better fix than that particular band-aid. Can we hold off on checking in a patch for now?
TBH I don't understand why it should crash, and therefore how your patch helps. Trying again using narrow strings should always work; indeed, the code did that before I touched it. Can you describe how it crashes?
(p.s. Considering that I can't test on Windows myself, I'm pretty happy that the code works as well as it does!)

> TBH I don't understand why it should crash, and therefore how your patch 
> helps. Trying again using narrow strings should always work; indeed, the 
> code did that before I touched it. Can you describe how it crashes?
The important part of the patch is the removal of the "!" in 
 if (!utime_read_time_arguments(&ua)) {
Without that change, if utime_read_time_arguments(&ua) fails then the unicode path is wrongly chosen. Then PyUnicode_AsUnicode(upath) is called when upath has not been initialized.

Without the check for RuntimeError
 os.utime("foo", times=(5,5), ns=(5,5))
raises
 TypeError("TypeError: 'str' does not support the buffer interface")
because we have fallen through to the narrow path. The correct error is
 RuntimeError("utime: you may specify either 'times' or 'ns' but not both")

Let me recap, just to make sure I have it straight. There are two errors on Windows:
* The ! on (what is currently) line 3770 is wrong:
 if (!utime_read_time_arguments(&ua)) {
* If you pass in a Unicode string but also pass in both times and ns,
 you get the wrong error: effectively it's complaining that the string
 is not narrow, when it should be complaining about having both times
 and ns.
For the former, obviously removing the ! is correct. But I like your idea of making the utime_read_time_argument() return value tell you what happened; that's what the Windows code needs to know.
So here it is! Please see the attached patch.

> Let me recap, just to make sure I have it straight. There are two errors 
> on Windows:
That's right. The patch looks good and passes for me on Windows.

New changeset fc5d2f4291ac by Larry Hastings in branch 'default':
Issue #14127: Fix two bugs with the Windows implementation.
http://hg.python.org/cpython/rev/fc5d2f4291ac 

There is another problem causing a fatal error in test_posix on Unix.
The attached patch fixes it: *ua->path should be decrefed not ua->path.

Looks good to me. You're a core contributor, yes? If not let me know and I'll commit it.
Though I must admit I'm baffled how I haven't seen that crash. I've run the unit tests a zillion times on this patch.

> Looks good to me. You're a core contributor, yes? If not let me know and 
> I'll commit it.
I will commit.
> Though I must admit I'm baffled how I haven't seen that crash. I've run 
> the unit tests a zillion times on this patch.
Were you running test_posix or only test_os?

By "the unit tests" I meant I ran the whole suite: Lib/test/regrtest.py. I know that runs test_os, and I assume it runs test_posix too.
I just ran test_posix by hand and it passed.
I'm developing on Linux (64-bit) in case that helps.

> I'm developing on Linux (64-bit) in case that helps.
I tested it on 32 bit Linux.
I have committed it, but I forgot to put the issue number in the commit message.

Revision 4deb7c1f49b9 

New changeset 709850f1ec67 by Larry Hastings in branch 'default':
Update Misc/NEWS for issues #14127 and #14705. (And, technically, #10148.)
http://hg.python.org/cpython/rev/709850f1ec67 

New changeset 05274ab06182 by Larry Hastings in branch 'default':
Update Misc/NEWS for issues #14127 and #14705. (And, technically, #10148.)
http://hg.python.org/cpython/rev/05274ab06182 

Closing this, as I have now removed os.utimensat and os.futimesat. As well as os.futimens, os.futimes, and os.lutimes. And in fact retooled os.utime in a pretty major way. (See #14626.)