ORC-1671: update timestamp doc in specfication - #1867

According to implementation:

Also, I'm not certain if the writer timezone is mandatory in the stripe if a TIMESTAMP column is present in the file. Might need some clarification on this?

IIUC, writer timezone field is mandatory if there is any timestamp type:

Thanks, I think I'll add this to the spec as well (maybe proto too, in the other repo?)

According to

I'm kinda unclear on how this exactly works, I'm just going off the C++ implementation here:

Happy to be corrected if my understanding or wording is inaccurate anywhere 👍

Basically here it tries to mimic the behavior on the Java side: https://github.com/apache/orc/blob/main/java/core/src/java/org/apache/orc/impl/TreeReaderFactory.java#L1280-L1295

Similarly, I'm not inclined to add these details to the specs if we are not 100% sure about it.

Usually
both writer and reader timezones default to UTC, however older ORC files may contain non-UTC writer timezones`

The main purpose of this timestamp type is to restore the wall clock time regardless of the reader time zone. IIRC, the Java implementation uses the writer local time zone instead of UTC. Due to insufficient C++ time zone utility support in earlier days, the C++ code by default uses UTC as the writer time zone to avoid complexity.

I see. I was confused as from the discussion/comments here: apache/arrow#34591

It suggested that the reader's timezone was important when deserializing the values

IMHO, we'd better not expose the implementation detail here. The original summary and the table below well explains what to expect from users.

I'm not sure I agree with that, as this timestamp type has been a great source of confusion for me when attempting to implement an ORC to Arrow reader in Rust.

For example, in the test file TestOrcFile.testDate2038.orc, according to PyArrow (which uses the ORC C++ reader underneath), the first value is:

>> from pyarrow import orc
>>> orc.read_table("/home/jeffrey/Code/orc/examples/TestOrcFile.testDate2038.orc")[0][0]
<pyarrow.TimestampScalar: '2038-05-05T12:34:56.100000000'>

The underlying integer value for the DATA stream is 736_601_696 seconds (let's ignore nanoseconds). When adding this to 1 Jan 2015 00:00:00, regardless of timezone, we get 5 May 2038 11:34:56. Which seems... surprising.

• https://www.timeanddate.com/date/dateadded.html?d1=1&m1=1&y1=2015&type=add&ay=&am=&aw=&ad=&h1=0&i1=0&s1=0&ah=&ai=&as=736601696&rec=

This isn't at all what I would expect hence my confusion (perhaps I just don't understand properly how DST factors into this). But diving into the internals and seeing reader & writer timezone offsets being considered when decoding the timestamp value just adds to the confusion, especially as that contradicts the specification.

I was hoping to open up some discussion on clarifying the exact definition for this timestamp type as part of this PR.

Sorry for the late reply. If you find any inconsistent behavior between C++ and Java sides, please refer to the Java implementation as reference. We need to fix any issue on the C++ side.

I'll try check more into this to confirm my understanding 👍

Moving to draft in the meantime