29
78
Fork
You've already forked r7rs
2

Source of random bits #71

Open
opened 2022年04月14日 13:37:08 +02:00 by mnw · 46 comments
Member
Copy link

In order to make a good and fast portable implementation of something like (srfi 27) possible, the Foundations will have to include some primitive procedures that produce sequences of random numbers and offer some source of entropy if available.

In order to make a good and fast portable implementation of something like `(srfi 27)` possible, the Foundations will have to include some primitive procedures that produce sequences of random numbers and offer some source of entropy if available.
Member
Copy link

If entropy is to be included, this will involve some participation of Committee E. In particular, questions of blocking would need to be addressed. I would like to see the Foundation include simple ways of getting a random integer and a random float, along with a no-arg randomize procedure; these are incredibly important in introductory teaching, in my experience.

If entropy is to be included, this will involve some participation of Committee E. In particular, questions of blocking would need to be addressed. I would like to see the Foundation include simple ways of getting a random integer and a random float, along with a no-arg randomize procedure; these are incredibly important in introductory teaching, in my experience.
Owner
Copy link

I think this falls entirely within the purview of Committee E. (On Unix systems, this can even be addressed in the small language by (open-binary-input-file "/dev/urandom").)

Relevant documents from WG1:

I think this falls entirely within the purview of Committee E. (On Unix systems, this can even be addressed in the small language by `(open-binary-input-file "/dev/urandom")`.) Relevant documents from WG1: - [RandomCowan](https://small.r7rs.org/wiki/RandomCowan/2/) - [RandomnessArcfide](https://small.r7rs.org/wiki/RandomnessArcfide/2/) - [RandomnessCommonLisp](https://small.r7rs.org/wiki/RandomnessCommonLisp/)
Author
Member
Copy link

I think this falls entirely within the purview of Committee E. (On Unix systems, this can even be addressed in the small language by (open-binary-input-file "/dev/urandom").)

This would not be portable, so encapsulating at least the most basic functionality in the Foundations can make sense.

Apart from that, the Foundations themselves may need random numbers, e.g. for UUIDs. Given that one goal of the foundation is to be self-contained, not all functionality can be outsourced to E. Given that the Foundations should form a practical languages by themselves, leaving out random numbers altogether may fail its goals.

I agree that everything fancy should be in B and E. This would include precise control over the entropy source

> I think this falls entirely within the purview of Committee E. (On Unix systems, this can even be addressed in the small language by `(open-binary-input-file "/dev/urandom")`.) This would not be portable, so encapsulating at least the most basic functionality in the Foundations can make sense. Apart from that, the Foundations themselves may need random numbers, e.g. for UUIDs. Given that one goal of the foundation is to be self-contained, not all functionality can be outsourced to E. Given that the Foundations should form a practical languages by themselves, leaving out random numbers altogether may fail its goals. I agree that everything fancy should be in B and E. This would include precise control over the entropy source

This would not be portable

Portable pseudo-random numbers are straightforward.

the Foundations themselves may need random numbers, e.g. for UUIDs.

Can you explain why the Foundation would require UUIDs?

> This would not be portable Portable pseudo-random numbers are straightforward. > the Foundations themselves may need random numbers, e.g. for UUIDs. Can you explain why the Foundation would require UUIDs?
Author
Member
Copy link

This would not be portable

Portable pseudo-random numbers are straightforward.

Yes but there's still the question of efficiency. I have to study the comments at the end of SRFI 27 about random number generation in more detail. It can very well be the case that a few lines of non-portable machine code yield a much faster the generator than one written in portable Scheme.

Anyway, the issue with getting entropy for the seed would remain.

the Foundations themselves may need random numbers, e.g. for UUIDs.

Can you explain why the Foundation would require UUIDs?

There are a few possible needs for UUIDs:

  • Read/write invariance of (pseudo-)gensyms. (See #70.)
  • Generation for UUIDs for non-generative records. This is needed at least for compatibility with R6RS, but also for read/write invariance of nongenerative record types. The virtue of nongenerative record types should be clear.
  • Serialization of syntax objects (carrying marks), for example for AOT expansion of libraries. The Foundations should be able to support a metacircular implementation. This is also the reason why Unsyntax needs SRFI 27 from the underlying R7RS host.

There are probably more possible needs I don't have on my mind now. And @vmanis mentioned a pedagocical one by that even a small language should include some procedures to produce randoms. (This is also mentioned in the linked WG1 papers, @dpk cited.)

In all these cases, a cryptographically secure RNG is not needed, but at least a seed (of 128 bits to be on the safe side, say), needs to be worked out randomly.

> > This would not be portable > > Portable pseudo-random numbers are straightforward. Yes but there's still the question of efficiency. I have to study the comments at the end of SRFI 27 about random number generation in more detail. It can very well be the case that a few lines of non-portable machine code yield a much faster the generator than one written in portable Scheme. Anyway, the issue with getting entropy for the seed would remain. > > > the Foundations themselves may need random numbers, e.g. for UUIDs. > > Can you explain why the Foundation would require UUIDs? There are a few possible needs for UUIDs: - Read/write invariance of (pseudo-)gensyms. (See #70.) - Generation for UUIDs for non-generative records. This is needed at least for compatibility with R6RS, but also for read/write invariance of nongenerative record types. The virtue of nongenerative record types should be clear. - Serialization of syntax objects (carrying marks), for example for AOT expansion of libraries. The Foundations should be able to support a metacircular implementation. This is also the reason why Unsyntax needs SRFI 27 from the underlying R7RS host. There are probably more possible needs I don't have on my mind now. And @vmanis mentioned a pedagocical one by that even a small language should include some procedures to produce randoms. (This is also mentioned in the linked WG1 papers, @dpk cited.) In all these cases, a cryptographically secure RNG is not needed, but at least a seed (of 128 bits to be on the safe side, say), needs to be worked out randomly.
Owner
Copy link

I am not convinced by any of the use cases for UUIDs, especially not for generating them automatically in the foundation libraries. See #70 for my response re gensyms. UUIDs for non-generative records are only needed if non-generative records are available syntactically, in which case they should be written in the source code and not generated at expand time. Serialization of syntax objects does not need them to actually carry marks — new ones can be generated at library load time.

I am not convinced by any of the use cases for UUIDs, especially not for generating them automatically in the foundation libraries. See #70 for my response re gensyms. UUIDs for non-generative records are only needed if non-generative records are available syntactically, in which case they should be written in the source code and not generated at expand time. Serialization of syntax objects does not need them to actually carry marks — new ones can be generated at library load time.
Author
Member
Copy link

I am not convinced by any of the use cases for UUIDs, especially not for generating them automatically in the foundation libraries. See #70 for my response re gensyms. UUIDs for non-generative records are only needed if non-generative records are available syntactically, in which case they should be written in the source code and not generated at expand time.

Why do you want to enforce users to write down some UUID all the time??? I am using the R6RS record type API and if it forced me to choose UUIDs all the time, it would lose a lot of its practicability.

Serialization of syntax objects does not need them to actually carry marks — new ones can be generated at library load time.

No, because the identity of marks must be preserved as two "communicating" syntax objects may be serialized at different times, even in different processes.

There may be other approaches, but UUIDs have the advantage that they are very easy and simple one.

> I am not convinced by any of the use cases for UUIDs, especially not for generating them automatically in the foundation libraries. See #70 for my response re gensyms. UUIDs for non-generative records are only needed if non-generative records are available syntactically, in which case they should be written in the source code and not generated at expand time. Why do you want to enforce users to write down some UUID all the time??? I am using the R6RS record type API and if it forced me to choose UUIDs all the time, it would lose a lot of its practicability. > Serialization of syntax objects does not need them to actually carry marks — new ones can be generated at library load time. No, because the identity of marks must be preserved as two "communicating" syntax objects may be serialized at different times, even in different processes. There may be other approaches, but UUIDs have the advantage that they are very easy and simple one.
Owner
Copy link

No, because the identity of marks must be preserved as two "communicating" syntax objects may be serialized at different times, even in different processes.

Communicating syntax objects in different processes? You’ve lost me.

> No, because the identity of marks must be preserved as two "communicating" syntax objects may be serialized at different times, even in different processes. Communicating syntax objects in different processes? You’ve lost me.
Owner
Copy link

Also, if gensyms are interned anyway, why not just make them regular symbols with the UUID part of their name? (string->symbol (string-append "gensym-" (uuid)))

Also, if gensyms are interned anyway, why not just make them regular symbols with the UUID part of their name? `(string->symbol (string-append "gensym-" (uuid)))`
Author
Member
Copy link

Also, if gensyms are interned anyway, why not just make them regular symbols with the UUID part of their name? (string->symbol (string-append "gensym-" (uuid)))

See one of my suggestions for #70. It would still make sense to wrap the above into a procedure gensym because this would allow an implementation to generate the name lazily, which can be quite beneficial depending on the speed of the uuid generator. In any case, the benefit of this approach would be that the range of Scheme core types wouldn't be increased, nor would their semantics. (The question and this answer belong to #70.)

> Also, if gensyms are interned anyway, why not just make them regular symbols with the UUID part of their name? `(string->symbol (string-append "gensym-" (uuid)))` See one of my suggestions for #70. It would still make sense to wrap the above into a procedure `gensym` because this would allow an implementation to generate the name lazily, which can be quite beneficial depending on the speed of the uuid generator. In any case, the benefit of this approach would be that the range of Scheme core types wouldn't be increased, nor would their semantics. (The question and this answer belong to #70.)
Author
Member
Copy link

No, because the identity of marks must be preserved as two "communicating" syntax objects may be serialized at different times, even in different processes.

Communicating syntax objects in different processes? You’ve lost me.

An AOT expansion of a library potentially contains syntax objects, which can carry marks. Consider two independent AOT compilations of, say, (scheme foo) and (scheme bar). Both libraries in turn import a third library, (scheme quux). If later, aa program imports both (scheme foo) and (scheme bar) the marks in the expansion of (scheme foo) that originally come from (scheme quux) must be compatible with the corresponding marks in the expansion of (scheme bar).

> > No, because the identity of marks must be preserved as two "communicating" syntax objects may be serialized at different times, even in different processes. > > Communicating syntax objects in different processes? You’ve lost me. An AOT expansion of a library potentially contains syntax objects, which can carry marks. Consider two independent AOT compilations of, say, `(scheme foo)` and `(scheme bar)`. Both libraries in turn import a third library, `(scheme quux)`. If later, aa program imports both `(scheme foo)` and `(scheme bar)` the marks in the expansion of `(scheme foo)` that originally come from `(scheme quux)` must be compatible with the corresponding marks in the expansion of `(scheme bar)`.
Owner
Copy link

If UUIDs or similar is used to implement that, marking and thus compilation is non-deterministic and builds cannot be reproducible.

A mark can simply be a counter associated with a library (‘the 0th mark applied while expanding (foo)’, ‘the 271th mark applied while expanding (bar)’ etc) which is simpler and deterministic.

If UUIDs or similar is used to implement that, marking and thus compilation is non-deterministic and builds cannot be reproducible. A mark can simply be a counter associated with a library (‘the 0th mark applied while expanding `(foo)`’, ‘the 271th mark applied while expanding `(bar)`’ etc) which is simpler and deterministic.
Author
Member
Copy link

As I wrote, there are other approaches, but they are less simple than UUIDs when you
don't need reproducible builds. (A simple gensym is certainly simpler than working with counters when you have to pair them together with other entities. And even, if not.)

Of course, you may still want to use counters for the code you write; others will want to use UUIDs for that. As we are not talking about the small language, there's no reason not to satisfy both groups of programmers and the implementation cost is very cheap. So I have the feeling we are spending more time here than would be needed to write down a spec and an implementation. :) ([3])

(In general, because of procedural macros, Scheme library builds aren't reproducible anyway; each time a library is built, it should include a unique version [1]; see [2].) Implementing the counter approach in a reproducible fashion is non-trivial as Scheme neither guarantees the order of processing in map or in lists of arguments.

--
[1] This, by the way, is another use case for UUIDs. Alternatively, it could be a hash over the library contents if you want to enforce reproducibility in a particular case.
[2] https://dl.acm.org/doi/10.1145/1291151.1291197
[3] And for R6RS compatibility, at least the implementation would have to be able to generate UUIDs anyway.

As I wrote, there are other approaches, but they are less simple than UUIDs when you don't need reproducible builds. (A simple gensym is certainly simpler than working with counters when you have to pair them together with other entities. And even, if not.) Of course, you may still want to use counters for the code you write; others will want to use UUIDs for that. As we are not talking about the small language, there's no reason not to satisfy both groups of programmers and the implementation cost is very cheap. So I have the feeling we are spending more time here than would be needed to write down a spec and an implementation. :) ([3]) (In general, because of procedural macros, Scheme library builds aren't reproducible anyway; each time a library is built, it should include a unique version [1]; see [2].) Implementing the counter approach in a reproducible fashion is non-trivial as Scheme neither guarantees the order of processing in `map` or in lists of arguments. -- [1] This, by the way, is another use case for UUIDs. Alternatively, it could be a hash over the library contents if you want to enforce reproducibility in a particular case. [2] https://dl.acm.org/doi/10.1145/1291151.1291197 [3] And for R6RS compatibility, at least the implementation would have to be able to generate UUIDs anyway.

Yes but there's still the question of efficiency.

This is a category mistake, I believe. Committee B requires only that a portable implementation exist, not that it be the most efficient possible implementation. R7RS implementers are free to provide non-portable solutions if efficiency is what they want.

Another example of this is SRFI 151, which has a portable but inefficient implementation of the core bitwise operations, but suggests that implementers provide their own non-portable implementations of them. That does not make it not a Committee B library.

In particular, questions of blocking would need to be addressed.

/dev/urandom doesn't block, and it's false to believe that it has worse randomness properties than /dev/random; see "Myths about /dev/urandom". The only issue is just after boot on Linux systems, and even that has been fixed.

Given that the Foundations should form a practical languages by themselves

I think this needs a separate discussion.

> Yes but there's still the question of efficiency. This is a category mistake, I believe. Committee B requires only that a portable implementation exist, not that it be the most efficient possible implementation. R7RS implementers are free to provide non-portable solutions if efficiency is what they want. Another example of this is SRFI 151, which has a portable but inefficient implementation of the core bitwise operations, but suggests that implementers provide their own non-portable implementations of them. That does not make it not a Committee B library. > In particular, questions of blocking would need to be addressed. `/dev/urandom` doesn't block, and it's false to believe that it has worse randomness properties than `/dev/random`; see ["Myths about /dev/urandom"](https://www.2uo.de/myths-about-urandom/). The only issue is just after boot on Linux systems, and even that has been fixed. > Given that the Foundations should form a practical languages by themselves I think this needs a separate discussion.
Author
Member
Copy link

Yes but there's still the question of efficiency.

This is a category mistake, I believe. Committee B requires only that a portable implementation exist, not that it be the most efficient possible implementation. R7RS implementers are free to provide non-portable solutions if efficiency is what they want.

Another example of this is SRFI 151, which has a portable but inefficient implementation of the core bitwise operations, but suggests that implementers provide their own non-portable implementations of them. That does not make it not a Committee B library.

You probably meant to write Committee F Library in your last sentence.

Given the total size of the final large libraries, many implementers or users will just use (or partially use) the sample implementations. Therefore, being able to have efficient sample implemementations is important. This applies, for example, to SRFI 151, which you cited, but also to SRFI 125 (and eq?-based hash tables in particular).

This doesn't mean that (scheme hash-table) or (scheme bitwise) would become part of the Foundations, but the Foundations should export enough procedures in some Foundation library disjoint from the large libraries on hash tables or bitwise operations to make reasonbly efficient implementations of the large procedures and syntax on hash tables and bitwise operations possible.

Similarly, the idea of this issue is not that the Foundations include (scheme random), a repackaged SRFI 27, - which would be a Committee B issue - but that they include a minimal set of two or three procedures in a disjoint namespace, say, (foundation ...) or (rnrs (7) ...), on which a fast SRFI 27 implementation could be built.

For a Scheme implementer seeking a reasonbly efficient large implementation, this would mean that they just have to pay attention to the Foundation procedures and could then just drop in the sample implementations of the large libraries.

> > Yes but there's still the question of efficiency. > > This is a category mistake, I believe. Committee B requires only that a portable implementation exist, not that it be the most efficient possible implementation. R7RS implementers are free to provide non-portable solutions if efficiency is what they want. > > Another example of this is SRFI 151, which has a portable but inefficient implementation of the core bitwise operations, but suggests that implementers provide their own non-portable implementations of them. That does not make it not a Committee B library. You probably meant to write *Committee F Library* in your last sentence. Given the total size of the final large libraries, many implementers or users will just use (or partially use) the sample implementations. Therefore, being able to have efficient sample implemementations is important. This applies, for example, to SRFI 151, which you cited, but also to SRFI 125 (and eq?-based hash tables in particular). This doesn't mean that `(scheme hash-table)` or `(scheme bitwise)` would become part of the Foundations, but the Foundations should export enough procedures in some Foundation library disjoint from the large libraries on hash tables or bitwise operations to make reasonbly efficient implementations of the large procedures and syntax on hash tables and bitwise operations possible. Similarly, the idea of this issue is not that the Foundations include `(scheme random)`, a repackaged SRFI 27, - which would be a Committee B issue - but that they include a minimal set of two or three procedures in a disjoint namespace, say, `(foundation ...)` or `(rnrs (7) ...)`, on which a fast SRFI 27 implementation could be built. For a Scheme implementer seeking a reasonbly efficient large implementation, this would mean that they just have to pay attention to the Foundation procedures and could then just drop in the sample implementations of the large libraries.

You probably meant to write Committee F Library in your last sentence.

I did not. However, the double negation makes the sentence hard to understand, and I reword it as follows: The fact that portable implementations of the core functions of SRFI 151 are slow does not remove SRFI 151 from the purview of Committee B, since they can be and are implemented portably using the Committee F+E language, which is all that Committee B requires in a technical sense: Committee B's chief concerns are non-technical.

Therefore, being able to have efficient sample implemementations is important.

I agree. However, being able to have efficient sample implementations is not the same as actually having them on hand. Commmittee B requires only that a portable implementation is available. But I have added this note about efficient implementations to the Committee B charter:

All libraries that pass Committee B will have portable implementations: if they are efficient, so much the better. However, a portable but less efficient implementation is acceptable, particularly if it contains clear indications of what the programmer thinks is important to optimize.

> You probably meant to write Committee F Library in your last sentence. I did not. However, the double negation makes the sentence hard to understand, and I reword it as follows: The fact that portable implementations of the core functions of SRFI 151 are slow does not remove SRFI 151 from the purview of Committee B, since they can be and are implemented portably using the Committee F+E language, which is all that Committee B requires in a technical sense: Committee B's chief concerns are non-technical. > Therefore, being able to have efficient sample implemementations is important. I agree. However, being *able* to have efficient sample implementations is not the same as actually having them on hand. Commmittee B requires only that a portable implementation is available. But I have added this note about efficient implementations to the Committee B charter: All libraries that pass Committee B will have portable implementations: if they are efficient, so much the better. However, a portable but less efficient implementation is acceptable, particularly if it contains clear indications of what the programmer thinks is important to optimize.
Author
Member
Copy link

You probably meant to write Committee F Library in your last sentence.

I did not. However, the double negation makes the sentence hard to understand, and I reword it as follows: The fact that portable implementations of the core functions of SRFI 151 are slow does not remove SRFI 151 from the purview of Committee B, since they can be and are implemented portably using the Committee F+E language, which is all that Committee B requires in a technical sense: Committee B's chief concerns are non-technical.

Actually, your sentence wasn't too hard to understand if read carefully. I somehow managed to overlook the "not" on my first read.

Anyway, I agree with you. The API and all the rest of the library (scheme bitwise) is for Committee B.

It still makes sense for the Foundations to offer at least some bitwise procedures to allow an efficient sample implementation for (scheme bitwise), but that wouldn't be a must.

(In the particular case of bitwise operations, there will probably be such an offer in the form of (rnrs arithmetic bitwise (7)) as part of an R6RS compatibility layer.)

Therefore, being able to have efficient sample implemementations is important.

I agree. However, being able to have efficient sample implementations is not the same as actually having them on hand. Commmittee B requires only that a portable implementation is available. But I have added this note about efficient implementations to the Committee B charter:

All libraries that pass Committee B will have portable implementations: if they are efficient, so much the better. However, a portable but less efficient implementation is acceptable, particularly if it contains clear indications of what the programmer thinks is important to optimize.

This looks good to me.

A particular example that comes to my mind is car+cdr from SRFI 1/(scheme list). Although it may be implemented more efficiently than what can be done portable, it doesn't necessarily make sense to add such a procedure to the Foundations.

> > You probably meant to write Committee F Library in your last sentence. > > I did not. However, the double negation makes the sentence hard to understand, and I reword it as follows: The fact that portable implementations of the core functions of SRFI 151 are slow does not remove SRFI 151 from the purview of Committee B, since they can be and are implemented portably using the Committee F+E language, which is all that Committee B requires in a technical sense: Committee B's chief concerns are non-technical. Actually, your sentence wasn't too hard to understand if read carefully. I somehow managed to overlook the "not" on my first read. Anyway, I agree with you. The API and all the rest of the library `(scheme bitwise)` is for Committee B. It still makes sense for the Foundations to offer at least some bitwise procedures to allow an efficient sample implementation for `(scheme bitwise)`, but that wouldn't be a must. (In the particular case of bitwise operations, there will probably be such an offer in the form of `(rnrs arithmetic bitwise (7))` as part of an R6RS compatibility layer.) > > Therefore, being able to have efficient sample implemementations is important. > > I agree. However, being *able* to have efficient sample implementations is not the same as actually having them on hand. Commmittee B requires only that a portable implementation is available. But I have added this note about efficient implementations to the Committee B charter: > > All libraries that pass Committee B will have portable implementations: if they are efficient, so much the better. However, a portable but less efficient implementation is acceptable, particularly if it contains clear indications of what the programmer thinks is important to optimize. This looks good to me. A particular example that comes to my mind is `car+cdr` from SRFI 1/`(scheme list)`. Although it may be implemented more efficiently than what can be done portable, it doesn't necessarily make sense to add such a procedure to the Foundations.

I somehow managed to overlook the "not" on my first read.

"Multiplex Negatio Ferblondiat"

> I somehow managed to overlook the "not" on my first read. ["Multiplex Negatio Ferblondiat"](http://itre.cis.upenn.edu/~myl/languagelog/archives/004716.html)

A version 1 UUID, rather than being mostly random bits like a version 4 UUID, has an internal structure that can be adapted to generating UUIDs efficiently without real-time access to high-quality randomness. All UUIDs are 128 bits in size, and have 6 bits of overhead indicating how the UUID was created. The remaining 122 bits of a version 4 UUID are cryptographically random.

In a version 1 UUID, however, there is a 48-bit node number (originally an Ethernet MAC address) representing the machine where the UUID is generated, a 60-bit timestamp (a count of 100-ns clunks since 1582年10月15日T00:00:00Z), and a 12-bit clock sequence number which is incremented whenever the machine cannot be certain that the clock was not set back or the node number changed. However, the node number can be random (in which case it can be set when the process starts; there is a bit to indicate whether it's a real MAC address or not), and the timestamp can be a simple count of how many UUIDs have been issued.

A version 1 UUID, rather than being mostly random bits like a version 4 UUID, has an internal structure that can be adapted to generating UUIDs efficiently without real-time access to high-quality randomness. All UUIDs are 128 bits in size, and have 6 bits of overhead indicating how the UUID was created. The remaining 122 bits of a version 4 UUID are cryptographically random. In a version 1 UUID, however, there is a 48-bit node number (originally an Ethernet MAC address) representing the machine where the UUID is generated, a 60-bit timestamp (a count of 100-ns clunks since 1582年10月15日T00:00:00Z), and a 12-bit clock sequence number which is incremented whenever the machine cannot be certain that the clock was not set back or the node number changed. However, the node number can be random (in which case it can be set when the process starts; there is a bit to indicate whether it's a real MAC address or not), and the timestamp can be a simple count of how many UUIDs have been issued.
Owner
Copy link

Bringing this back to the original idea: any source of randomness has to specify what the randomness is good for. What guarantees we want to make in this area determine, in practice, which volume this ends up in.

Cryptography has very high and particular standards for randomness, both in terms of predictability of results and in terms of timing. A guarantee of cryptographic-quality randomness means the implementation will have to ask the OS for entropy. This means it is an Environments issue.

If we can get away with randomness good enough for miscellaneous statistical purposes (such as generating UUIDs with a guarantee close to no collisions for 264 generations), an implementation in pure Scheme is reasonable. That makes this a Batteries issue.

I can’t think of a situation where it would become a Foundations issue from the point of view of the spec, even if a particular implementation uses randomness somewhere.

Bringing this back to the original idea: any source of randomness has to specify what the randomness is good for. What guarantees we want to make in this area determine, in practice, which volume this ends up in. Cryptography has very high and particular standards for randomness, both in terms of predictability of results and in terms of timing. A guarantee of cryptographic-quality randomness means the implementation will have to ask the OS for entropy. This means it is an Environments issue. If we can get away with randomness good enough for miscellaneous statistical purposes (such as generating UUIDs with a guarantee close to no collisions for 2<sup>64</sup> generations), an implementation in pure Scheme is reasonable. That makes this a Batteries issue. I can’t think of a situation where it would become a Foundations issue from the point of view of the spec, even if a particular implementation uses randomness somewhere.
dpk removed this from the Valued Fascicle milestone 2023年08月29日 20:15:50 +02:00
Member
Copy link

I can’t think of a situation where it would become a Foundations issue from the point of view of the spec, even if a particular implementation uses randomness somewhere.

I would like to see basic randomness in Foundations (as a former teacher, I always wanted to introduce random number generation pretty early in a CS1 course), with no guarantee of quality (other than "best efforts", perhaps). That would mean random-int, random-float, and randomize!). I do believe that a Foundations-only Scheme implementation should be usable for teaching purposes. Batteries could add various distributions, etc. I would strongly recommend not ever specifying quality, other than stating that implementers should include a statement of the quality.

Batteries could provide a version of randomize! that used entropy in an unspecified way; this would work on (at least) Linux, BSD, MacOS, and Windows, and would raise an error if no entropy was available (an error is less ignorable than returning a success boolean).

Environments could then have further specification of the way(s) in which entropy is used.

>I can’t think of a situation where it would become a Foundations issue from the point of view of the spec, even if a particular implementation uses randomness somewhere. I would like to see basic randomness in Foundations (as a former teacher, I always wanted to introduce random number generation pretty early in a CS1 course), with no guarantee of quality (other than "best efforts", perhaps). That would mean random-int, random-float, and randomize!). I do believe that a Foundations-only Scheme implementation should be usable for teaching purposes. Batteries could add various distributions, etc. I would strongly recommend not ever specifying quality, other than stating that implementers should include a statement of the quality. Batteries could provide a version of randomize! that used entropy in an unspecified way; this would work on (at least) Linux, BSD, MacOS, and Windows, and would raise an error if no entropy was available (an error is less ignorable than returning a success boolean). Environments could then have further specification of the way(s) in which entropy is used.
Owner
Copy link

I would like to see a basic set of randomness primitives with minimal guarantees, and a differently named set designed for cryptographic randomness. Using different names would help to prevent the otherwise inevitable use of unsafe randomness for cryptographic applications.

I would like to see a basic set of randomness primitives with minimal guarantees, and a differently named set designed for cryptographic randomness. Using different names would help to prevent the otherwise inevitable use of unsafe randomness for cryptographic applications.
Author
Member
Copy link

I would like to see a basic set of randomness primitives with minimal guarantees, and a differently named set designed for cryptographic randomness. Using different names would help to prevent the otherwise inevitable use of unsafe randomness for cryptographic applications.

That sounds like a good idea to me. The Foundations would have to include a random bit source but would make no guarantees about cryptographic security. One implementation may access /dev/urandom, another implementation may just use clock_gettime.

> I would like to see a basic set of randomness primitives with minimal guarantees, and a differently named set designed for cryptographic randomness. Using different names would help to prevent the otherwise inevitable use of unsafe randomness for cryptographic applications. That sounds like a good idea to me. The Foundations would have to include a random bit source but would make no guarantees about cryptographic security. One implementation may access `/dev/urandom`, another implementation may just use `clock_gettime`.
Owner
Copy link

Following up on the earlier discussion, perhaps names beginning pseudo-random- would be appropriate for basic randomness procedures, whereas names beginning with crypto-random- would be for cryptographically sound randomness procedures.

Another approach would be to use the same names everywhere, but have an optional argument that specifies the quality of randomness expected, e.g. (random-integer 1024 'cryptographic). That would have the advantage that it would be hard for someone to miss the distinction.

Just thoughts.

Following up on the earlier discussion, perhaps names beginning `pseudo-random-` would be appropriate for basic randomness procedures, whereas names beginning with `crypto-random-` would be for cryptographically sound randomness procedures. Another approach would be to use the same names everywhere, but have an optional argument that specifies the quality of randomness expected, e.g. `(random-integer 1024 'cryptographic)`. That would have the advantage that it would be hard for someone to miss the distinction. Just thoughts.
Owner
Copy link

In any case, I would recommend that Foundations only specify the less strict guarantees on randomness, leaving the way open for a later fascicle to add cryptographic randomness.

In any case, I would recommend that Foundations only specify the less strict guarantees on randomness, leaving the way open for a later fascicle to add cryptographic randomness.
Member
Copy link

I've written a proposal for a low-level RNG API here. It allows for entropy sources (/dev/random, a hash of the current time, etc.) and deterministic RNGs with serializable state (Mersenne Twister, etc.). It's user-extensible in a similar way to SRFI-225 dictionaries, where a random-source descriptor contains the procedures necessary to use the random number generator. Higher level APIs and non-uniform distributions like in SRFI 194 could be implemented on top of it.

No sample implementation yet.

I've written a proposal for a low-level RNG API [here](https://codeberg.org/phm/user-configurable-random). It allows for entropy sources (`/dev/random`, a hash of the current time, etc.) and deterministic RNGs with serializable state (Mersenne Twister, etc.). It's user-extensible in a similar way to SRFI-225 dictionaries, where a random-source descriptor contains the procedures necessary to use the random number generator. Higher level APIs and non-uniform distributions like in SRFI 194 could be implemented on top of it. No sample implementation yet.
Member
Copy link

Taylor Campbell had the following to say on #scheme back on 2025年12月01日. (Sorry, I didn't remember to post this here at the time.)

You should use cryptographic PRNGs by default, and let applications seeking higher performance at the cost of security opt into broken PRNGs. And eschew extra application-level options when querying a system entropy source for waffling about ‘really random’ vs ‘pseudo random’ vs ‘blocking random’ -- these are generally semantically incoherent and a gigantic waste of time. I'm a big fan of deterministic randomization, with the caveat that it entails a commitment to a particular PRNG algorithm. E.g., chacha-fast-key-erasure768 is a reasonable choice of deterministic algorithm today, mapping a seed and a sequence of request sizes to a sequence of answers. But maybe you don't want to commit to that under the generic name ‘random’. Whatever the default thing people will reach for should, on machines with enough entropy sources, (a) be seeded by an unpredictable secret, e.g. from getentropy(seed, 32) in POSIX.1-2024 systems, (b) generate output by a cryptographic PRNG under that seed.

Of course some machines don't have enough entropy sources, but that's a problem of building the machine, not of software that was written by someone else that you're trying to run on the machine. For example, run a VM on a host that intercepts the RDRAND instruction and computes it deterministically.

If, on the other hand, you create a variety of options like the Linux getrandom syscall's GRND_RANDOM, GRND_INSECURE, GRND_NONBLOCK, you just leave everyone confused and all of the options end up being problematic or broken or misused or all of the above.

Also: a random floating-point sampler should be supported on (0,1], not [0,1). (Really, it should be supported on [0,1], but for binary32 or larger, a result of 0 is so improbable that it is effectively impossible.)

Taylor Campbell had the following to say on #scheme back on 2025年12月01日. (Sorry, I didn't remember to post this here at the time.) > You should use cryptographic PRNGs by default, and let applications seeking higher performance at the cost of security opt into broken PRNGs. And eschew extra application-level options when querying a system entropy source for waffling about ‘really random’ vs ‘pseudo random’ vs ‘blocking random’ -- these are generally semantically incoherent and a gigantic waste of time. I'm a big fan of deterministic randomization, with the caveat that it entails a commitment to a particular PRNG algorithm. E.g., chacha-fast-key-erasure768 is a reasonable choice of deterministic algorithm today, mapping a seed and a sequence of request sizes to a sequence of answers. But maybe you don't want to commit to that under the generic name ‘random’. Whatever the default thing people will reach for should, on machines with enough entropy sources, (a) be seeded by an unpredictable secret, e.g. from getentropy(seed, 32) in POSIX.1-2024 systems, (b) generate output by a cryptographic PRNG under that seed. > > Of course some machines don't have enough entropy sources, but that's a problem of building the machine, not of software that was written by someone else that you're trying to run on the machine. For example, run a VM on a host that intercepts the RDRAND instruction and computes it deterministically. > > If, on the other hand, you create a variety of options like the Linux getrandom syscall's GRND_RANDOM, GRND_INSECURE, GRND_NONBLOCK, you just leave everyone confused and all of the options end up being problematic or broken or misused or all of the above. > > Also: a random floating-point sampler should be supported on (0,1], not [0,1). (Really, it should be supported on [0,1], but for binary32 or larger, a result of 0 is so improbable that it is effectively impossible.)
Member
Copy link

Reflecting on my own proposal some more:

We should differentiate between procedures that generate random bits, and procedures that takes those random bits and convert them into useful data structures.

Foundations should have procedures that convert already-taken random bytes into a fixnum and a flonum, because conversion into those in a uniform way depends on the representation of those types.

So the Foundations API would be

(random-bytes->fixnum bv)
bytes-for-fixnum
(random-bytes->flonum bv)
bytes-for-flonum

where bv is a bytevector of length bytes-for-fixnum and bytes-for-flonum respectively for those procedures. The procedures are constrained such that a uniformly sampled bv of the appropriate length will give a uniformly sampled fixnum/flonum.

Entropic randomness, CSPRNGs, fast non-cryptographic PRNGs, etc. will then be considered later in Environments and Batteries. I would like the APIs of RNGs to be uniform and user-configurable so that someone can swap in their own RNG like in my pre-SRFI, but I don't like the API of my pre-SRFI and would like something else.

Reflecting on my own proposal some more: We should differentiate between procedures that generate random bits, and procedures that takes those random bits and convert them into useful data structures. Foundations should have procedures that convert already-taken random bytes into a fixnum and a flonum, because conversion into those in a uniform way depends on the representation of those types. So the Foundations API would be ```scheme (random-bytes->fixnum bv) bytes-for-fixnum (random-bytes->flonum bv) bytes-for-flonum ``` where `bv` is a bytevector of length `bytes-for-fixnum` and `bytes-for-flonum` respectively for those procedures. The procedures are constrained such that a uniformly sampled `bv` of the appropriate length will give a uniformly sampled fixnum/flonum. Entropic randomness, CSPRNGs, fast non-cryptographic PRNGs, etc. will then be considered later in Environments and Batteries. I would like the APIs of RNGs to be uniform and user-configurable so that someone can swap in their own RNG like in my pre-SRFI, but I don't like the API of my pre-SRFI and would like something else.
Member
Copy link

A general purpose PRNG for both cryptographic and non-cryptographic uses is a bad idea because they target different users.

  1. Cryptographic PRNG users want cryptographic random numbers. If there is an issue with the implementation, or if there is a better algorithm that people want to use, then people will want a fix to the generator even if, given the same seed, the generator will generate different numbers.
  2. Non-cryptographic PRNGs will have people that want the same generated numbers every time. There is a much bigger risk that a change to the implementation will break code written with the expectation that certain seeds generate certain sequences.

So my get-entropy! procedure (likely a wrapper around getentropy(3) or arc4random(3)) is probably sufficient for "give me cryptographic bytes". A random-source API like in SRFI 27 is designed around consistency (which is the other part of my pre-SRFI). It could use a CSPRNG but is meant to be consistent across implementation updates.

(The arc4random API is also a good example of why you shouldn't try to specify what PRNG you're using. OpenBSD no longer uses RC4.)


John and I discussed this briefly during the meeting but I think portable seeding of RNGs is important. Given a fixnum, one could generate a sequence of random bytes suitable for seeding a more advanced PRNG. (See xoshiro256++, which recommends the use of splitmix64 seeded with any u64 to generate its seed as a sequence of random bytes.) The user should understand that seeding an RNG with 0 may not give cryptographic quality results, but implementations can be set up to generate reasonable quality randomness given a fixnum seed.


I am not familiar enough with the cases where high-entropy randomness is unavailable. @alaric mentioned some scenarios and I would like to hear about them. (If a procedure like get-entropy! is implemented in terms of OS interfaces, then getentropy fails with a non-specific error, and arc4random always succeeds. I am not familiar enough with Windows or phone interfaces to comment on what entropic random primitives they have.)

A general purpose PRNG for both cryptographic and non-cryptographic uses is a bad idea because they target different users. 1. Cryptographic PRNG users want cryptographic random numbers. If there is an issue with the implementation, or if there is a better algorithm that people want to use, then people will want a fix to the generator even if, given the same seed, the generator will generate different numbers. 2. Non-cryptographic PRNGs will have people that want the same generated numbers every time. There is a much bigger risk that a change to the implementation will break code written with the expectation that certain seeds generate certain sequences. So my `get-entropy!` procedure (likely a wrapper around [`getentropy(3)`](https://www.man7.org/linux/man-pages/man3/getentropy.3.html) or [`arc4random(3)`](https://man.openbsd.org/arc4random.3)) is probably sufficient for "give me cryptographic bytes". A random-source API like in SRFI 27 is designed around consistency (which is the other part of my pre-SRFI). It could use a CSPRNG but is meant to be consistent across implementation updates. (The `arc4random` API is also a good example of why you shouldn't try to specify what PRNG you're using. OpenBSD no longer uses RC4.) ____________________________________________________________ John and I discussed this briefly during the meeting but I think portable seeding of RNGs is important. Given a fixnum, one could generate a sequence of random bytes suitable for seeding a more advanced PRNG. (See [xoshiro256++](https://prng.di.unimi.it/xoshiro256plusplus.c), which recommends the use of [splitmix64](https://prng.di.unimi.it/splitmix64.c) seeded with any u64 to generate its seed as a sequence of random bytes.) The user should understand that seeding an RNG with `0` may not give cryptographic quality results, but implementations can be set up to generate reasonable quality randomness given a fixnum seed. ____________________________________________________________ I am not familiar enough with the cases where high-entropy randomness is unavailable. @alaric mentioned some scenarios and I would like to hear about them. (If a procedure like `get-entropy!` is implemented in terms of OS interfaces, then `getentropy` fails with a non-specific error, and `arc4random` always succeeds. I am not familiar enough with Windows or phone interfaces to comment on what entropic random primitives they have.)

I posted to #scheme about Isaac, which is a strong PRNG you seed with /dev/(u)random to make it a CSPRNG. CSPRNGs matter when you need best-quality randomness at high speeds. Are you making tens of thousands of encrypted connections per second and need to create session keys? Just reading from a random device isn't going to cut it.

Isaac is in CL, but it's only about 200 LoC and should be easy to transpose to the key of Scheme.

I posted to #scheme about Isaac, which is a strong PRNG you seed with `/dev/(u)random` to make it a CSPRNG. CSPRNGs matter when you need best-quality randomness at high speeds. Are you making tens of thousands of encrypted connections per second and need to create session keys? Just reading from a random device isn't going to cut it. [Isaac is in CL](https://github.com/hoytech/antiweb/blob/master/bundled/isaac.lisp), but it's only about 200 LoC and should be easy to transpose to the key of Scheme.
Member
Copy link

@johnwcowan wrote in #71 (comment):

Are you making tens of thousands of encrypted connections per second and need to create session keys? Just reading from a random device isn't going to cut it.

That sounds like a pretty specialized use-case. The standard libraries can't encompass every use of random bits. Someone who is doing that is likely linked to OpenSSL and may want to use their PRNG over the one in the implementation, which is likely to be less battle tested.

A standard, default interface in the implementation is going to be less flexible than something supplied by a user or by an external library. I can imagine a scenario where people go "don't use generate-crypto-bytes from (scheme random) because implementations X, Y, Z use a really old PRNG that hasn't been updated in forever."

The second part of my proposal is a user-configurable API for PRNGs, so that the programmer can use whatever PRNG they want and write their code using one API. (So if one wants to use Issac written in Scheme, or RAND_bytes written in OpenSSL, etc. they can.)

@johnwcowan wrote in https://codeberg.org/scheme/r7rs/issues/71#issuecomment-12012429: > Are you making tens of thousands of encrypted connections per second and need to create session keys? Just reading from a random device isn't going to cut it. That sounds like a pretty specialized use-case. The standard libraries can't encompass every use of random bits. Someone who is doing that is likely linked to OpenSSL and may want to use their PRNG over the one in the implementation, which is likely to be less battle tested. A standard, default interface in the implementation is going to be less flexible than something supplied by a user or by an external library. I can imagine a scenario where people go "don't use `generate-crypto-bytes` from `(scheme random)` because implementations X, Y, Z use a really old PRNG that hasn't been updated in forever." The second part of my proposal is a user-configurable API for PRNGs, so that the programmer can use whatever PRNG they want and write their code using one API. (So if one wants to use Issac written in Scheme, or `RAND_bytes` written in OpenSSL, etc. they can.)
Member
Copy link

I volunteered to summarise some of the issues around random number sources in this issue, but it turned into a long enough ramble that I posted it to the mailing list instead. However, to not break my promise and to make sure it's easily found again, here's the posting in the archives:

https://groups.google.com/g/scheme-reports-wg2/c/nMdFKe_dBbE

I volunteered to summarise some of the issues around random number sources in this issue, but it turned into a long enough ramble that I posted it to the mailing list instead. However, to not break my promise and to make sure it's easily found again, here's the posting in the archives: https://groups.google.com/g/scheme-reports-wg2/c/nMdFKe_dBbE
Member
Copy link

The Rust rand library uses its type system to denote which RNGs are cryptographic, and which ones can "fail" (i.e. for depleted entropy or whatever reasons) and which ones are "infallible" (always return numbers from their sequence). The experience there may be useful if we want to differentiate PRNG and CS(P)RNGs at the API level.

[The Rust `rand` library](https://docs.rs/rand/latest/rand/) uses its type system to denote which RNGs are cryptographic, and which ones can "fail" (i.e. for depleted entropy or whatever reasons) and which ones are "infallible" (always return numbers from their sequence). The experience there may be useful if we want to differentiate PRNG and CS(P)RNGs at the API level.
Member
Copy link

Here's my modified Foundations API for random numbers, after feedback from others:

(random-bytes->integer binary-input-port start end)

If binary-input-port generates a sequence of uniformly distributed random bytes, then this procedure will return an integer uniformly distributed between start and end.

NOTE: It cannot be known portably how many random bytes are required to generate a random integer.

(random-bytes->flonum binary-input-port start end)

If binary-input-port generates a sequence of uniformly distributed random bytes, returns a flonum drawn approximately from the uniform distribution of real numbers between start and end.

NOTE: It cannot be known portably how many random bytes are required to generate a random flonum.

RATIONALE: This API is intended for flonums that implement IEEE binary floating point arithmetic. There are known algorithms for drawing binary floating-point numbers from an interval. Many common methods are not as uniform as possible across arbitrary intervals: see Frédéric Goualard, Drawing random floating-point numbers from an interval , ACM Transactions on Modeling and Computer Simulation, 2022, 32 (3).


Other distributions, rational numbers, complex numbers, etc. along with specified generators can be in Batteries.

If an implementation provides non-flonum inexact reals (decimal floating point, for instance), they should also supply their own random bytes converters.

Here's my modified Foundations API for random numbers, after feedback from others: ```scheme (random-bytes->integer binary-input-port start end) ``` If `binary-input-port` generates a sequence of uniformly distributed random bytes, then this procedure will return an integer uniformly distributed between `start` and `end`. NOTE: It cannot be known portably how many random bytes are required to generate a random integer. ```scheme (random-bytes->flonum binary-input-port start end) ``` If `binary-input-port` generates a sequence of uniformly distributed random bytes, returns a flonum drawn approximately from the uniform distribution of real numbers between `start` and `end`. NOTE: It cannot be known portably how many random bytes are required to generate a random flonum. RATIONALE: This API is intended for flonums that implement IEEE binary floating point arithmetic. There are known algorithms for drawing binary floating-point numbers from an interval. Many common methods are not as uniform as possible across arbitrary intervals: see Frédéric Goualard, *[Drawing random floating-point numbers from an interval](https://hal.science/hal-03282794v2)*, ACM Transactions on Modeling and Computer Simulation, 2022, 32 (3). ___________________________ Other distributions, rational numbers, complex numbers, etc. along with specified generators can be in Batteries. If an implementation provides non-flonum inexact reals (decimal floating point, for instance), they should also supply their own random bytes converters.

@phm wrote in #71 (comment):

I like the names and most of the interface, but I assume the port is read a byte at a time based on:

NOTE: It cannot be known portably how many random bytes are required to generate a random integer.

If that's true, I still think a procedure to return a random byte is better as the first argument. It's very easy to write (lambda () (read-u8 binary-input-port) when you have an actual port, but we don't want to encourage people to draw potentially large numbers of potentially large values directly from /dev/random.

@phm wrote in https://codeberg.org/scheme/r7rs/issues/71#issuecomment-13134708: I like the names and most of the interface, but I assume the port is read a byte at a time based on: > NOTE: It cannot be known portably how many random bytes are required to generate a random integer. If that's true, I still think a procedure to return a random byte is better as the first argument. It's very easy to write `(lambda () (read-u8 `*binary-input-port*`)` when you have an actual port, but we don't want to encourage people to draw potentially large numbers of potentially large values directly from `/dev/random`.

After reading this paper on subnormal numbers, I wonder if there shouldn't be a way to exclude them, given that some hardware handles them with software emulation.

After reading [this paper on subnormal numbers](https://www.acsel-lab.com/arithmetic/arith16/papers/ARITH16_Schwarz.pdf), I wonder if there shouldn't be a way to exclude them, given that [some hardware handles them with software emulation](https://github.com/rust-lang/rust/issues/123123).
Member
Copy link

@johnwcowan wrote in #71 (comment):

but I assume the port is read a byte at a time based on

Not necessarily. The note is a notice to users not to expect that the binary sequence is incremented by a fixed amount every call. For instance, if the user requests an integer between 56 and 64 bits in size, the random integer procedure could request 7 bytes in a block.

More importantly, the conversion algorithm could use rejection sampling: Taylor Campbell's algorithm is one example. I don't know if Frédéric Goualard's algorithm does that, I have to study it more.


@johnwcowan wrote in #71 (comment):

After reading this paper on subnormal numbers, I wonder if there shouldn't be a way to exclude them, given that some hardware handles them with software emulation.

There isn't an intrinsic speed penalty to generating them because generator algorithms are usually bit-manipulation.

If you could detect a subnormal you might be able to rejection-sample them away. (I would have to think more about it.) This is a problem whose solution depends on if we require IEEE arithmetic, and we would also probably want to add procedure to support things like enabling floating-point modes.

@johnwcowan wrote in https://codeberg.org/scheme/r7rs/issues/71#issuecomment-13136331: > but I assume the port is read a byte at a time based on Not necessarily. The note is a notice to users not to expect that the binary sequence is incremented by a fixed amount every call. For instance, if the user requests an integer between 56 and 64 bits in size, the random integer procedure could request 7 bytes in a block. More importantly, the conversion algorithm could use rejection sampling: [Taylor Campbell's algorithm](https://mumble.net/~campbell/2014/04/28/uniform-random-float) is one example. I don't know if Frédéric Goualard's algorithm does that, I have to study it more. ___________________________ @johnwcowan wrote in https://codeberg.org/scheme/r7rs/issues/71#issuecomment-13137951: > After reading [this paper on subnormal numbers](https://www.acsel-lab.com/arithmetic/arith16/papers/ARITH16_Schwarz.pdf), I wonder if there shouldn't be a way to exclude them, given that [some hardware handles them with software emulation](https://github.com/rust-lang/rust/issues/123123). There isn't an intrinsic speed penalty to generating them because generator algorithms are usually bit-manipulation. If you could detect a subnormal you might be able to rejection-sample them away. (I would have to think more about it.) This is a problem whose solution depends on if we require IEEE arithmetic, and we would also probably want to add procedure to support things like enabling floating-point modes.

There isn't an intrinsic speed penalty to generating them

No, but there is a speed penalty for using them. Unpredictably having your code slow down by two orders of magnitude because you get a subnormal number would be disastrous. I would even say the default should be to never generate them, rather than requiring a wrapper to reject them (which is two floating-point comparisons involving the smallest (or largest negative) flonum.

if we require IEEE arithmetic

Over at #275 you were in favor of requiring flonums to always be binary64, and I agree. There are no chips (except microcontrollers) still being made that don't provide binary64 and binary32, and there are only a few Schemes (RacketBC, Kawa, NexJ) that even support binary32.

> There isn't an intrinsic speed penalty to generating them No, but there is a speed penalty for using them. Unpredictably having your code slow down by two orders of magnitude because you get a subnormal number would be disastrous. I would even say the default should be to never generate them, rather than requiring a wrapper to reject them (which is two floating-point comparisons involving the smallest (or largest negative) flonum. > if we require IEEE arithmetic Over at #275 you were in favor of requiring flonums to always be binary64, and I agree. There are no chips (except microcontrollers) still being made that don't provide binary64 and binary32, and there are only a few Schemes (RacketBC, Kawa, NexJ) that even support binary32.
Member
Copy link

@johnwcowan wrote in #71 (comment):

No, but there is a speed penalty for using them. Unpredictably having your code slow down by two orders of magnitude because you get a subnormal number would be disastrous. I would even say the default should be to never generate them, rather than requiring a wrapper to reject them (which is two floating-point comparisons involving the smallest (or largest negative) flonum.

Subnormals can be generated from arithmetic operations, however. If one is really concerned about them, then the standard/an implementation can supply a flush-to-zero mode (for example, Julia).

The RNG would have to be careful, as the subnormals being gone makes the range of 0 values bigger. But the implementation of the RNG should be able to query the subnormal mode.


Frédéric Goualard's algorithm allows for [a,b], [a,b), etc. This is a modification to the previous that would allow for that behavior. (Some generator algorithms would not be able to reach those, however. So maybe those flags would be advisory?)

(random-bytes->flonum binary-input-port start end [flags])

If binary-input-port generates a sequence of uniformly distributed random bytes, returns a flonum drawn approximately from the uniform distribution of real numbers between start exclusive and end exclusive.

flags is an association list (by default, the empty list). The flags are symbols and the following are supported:

  1. include-start?: Include the start bound (default false).
  2. include-end?: Include the end bound (default false).
@johnwcowan wrote in https://codeberg.org/scheme/r7rs/issues/71#issuecomment-13142415: > No, but there is a speed penalty for using them. Unpredictably having your code slow down by two orders of magnitude because you get a subnormal number would be disastrous. I would even say the default should be to never generate them, rather than requiring a wrapper to reject them (which is two floating-point comparisons involving the smallest (or largest negative) flonum. Subnormals can be generated from arithmetic operations, however. If one is really concerned about them, then the standard/an implementation can supply a flush-to-zero mode (for example, [Julia](https://docs.julialang.org/en/v1/manual/performance-tips/#Treat-Subnormal-Numbers-as-Zeros)). The RNG would have to be careful, as the subnormals being gone makes the range of `0` values bigger. But the implementation of the RNG should be able to query the subnormal mode. ______________________________ Frédéric Goualard's algorithm allows for `[a,b]`, `[a,b)`, etc. This is a modification to the previous that would allow for that behavior. (Some generator algorithms would not be able to reach those, however. So maybe those flags would be advisory?) ```scheme (random-bytes->flonum binary-input-port start end [flags]) ``` If binary-input-port generates a sequence of uniformly distributed random bytes, returns a flonum drawn approximately from the uniform distribution of real numbers between `start` exclusive and `end` exclusive. `flags` is an association list (by default, the empty list). The flags are symbols and the following are supported: 1. `include-start?`: Include the start bound (default false). 2. `include-end?`: Include the end bound (default false).
Member
Copy link

@phm wrote in #71 (comment):

Many common methods are not as uniform as possible across arbitrary intervals: see Frédéric Goualard, Drawing random floating-point numbers from an interval.

Thanks very much for linking to this paper. It's a sobering read. While I'm only part of the way through, it already seems clear to me that any standard Scheme random-number interface should include a flonum-in-arbitrary-interval procedure. If the flonum reader uses the common [0, 1) interval, people will no doubt use the equally common (and wrong) y = (b − a)x + a transformation to get to the desired interval.

@phm wrote in https://codeberg.org/scheme/r7rs/issues/71#issuecomment-13134708: > Many common methods are not as uniform as possible across arbitrary intervals: see Frédéric Goualard, _Drawing random floating-point numbers from an interval_. Thanks very much for linking to this paper. It's a sobering read. While I'm only part of the way through, it already seems clear to me that any standard Scheme random-number interface should include a flonum-in-arbitrary-interval procedure. If the flonum reader uses the common [0, 1) interval, people will no doubt use the equally common (and wrong) `y = (b − a)x + a` transformation to get to the desired interval.

This is best done using a small rebasing of SRFI 194, which already provides generators for arbitrary bounded real numbers, to work on top of SRFI 271 random ports instead of SRFI 27 random generators. Most of the implementation can remain intact.

It would be possible to re-engineer 194 to produce ports as well as consuming them, but only by generalizing ports to return arbitrary objects snd not just characters or bytes. Gambit currently uses this model, allowing things like directory ports, where you get the next filename by calling read, but IMHO this is too big a change.

This is best done using a small rebasing of SRFI 194, which already provides generators for arbitrary bounded real numbers, to work on top of SRFI 271 random ports instead of SRFI 27 random generators. Most of the implementation can remain intact. It would be possible to re-engineer 194 to produce ports as well as consuming them, but only by generalizing ports to return arbitrary objects snd not just characters or bytes. Gambit currently uses this model, allowing things like directory ports, where you get the next filename by calling `read`, but IMHO this is too big a change.
Member
Copy link

@johnwcowan wrote in #71 (comment):

This is best done using a small rebasing of SRFI 194, which already provides generators for arbitrary bounded real numbers, to work on top of SRFI 271 random ports instead of SRFI 27 random generators. Most of the implementation can remain intact.

I'm working on it. I'm not entirely sure how much of 194 should be included. (I don't really understand the more complicated generators.)

From what I can tell, the 194 sample implementation uses a variation on y = (b − a)x + a to generate floats in an arbitrary interval. There are probably other things to fix beyond that.

@johnwcowan wrote in https://codeberg.org/scheme/r7rs/issues/71#issuecomment-18796820: > This is best done using a small rebasing of SRFI 194, which already provides generators for arbitrary bounded real numbers, to work on top of SRFI 271 random ports instead of SRFI 27 random generators. Most of the implementation can remain intact. I'm working on it. I'm not entirely sure how much of 194 should be included. (I don't really understand the more complicated generators.) From what I can tell, the 194 sample implementation uses a variation on `y = (b − a)x + a` to generate floats in an arbitrary interval. There are probably other things to fix beyond that.
Member
Copy link

Personally I would rather an API that returns a single randomly selected object vs. returning a generator. For example (random-normal port mean deviation) instead of (make-normal-generator port mean deviation).

(削除) A generator is a closure allocation which is expensive when one wants to, say, fill a f64vector with random numbers, unless one has a Sufficiently Smart compiler. Creating a generator by wrapping a procedure that generates one number is easy. (削除ここまで)

The only benefit of a generator interface is when the implementation generates a block of random objects and wishes to buffer it.

Personally I would rather an API that returns a single randomly selected object vs. returning a generator. For example `(random-normal port mean deviation)` instead of `(make-normal-generator port mean deviation)`. ~~A generator is a closure allocation which is expensive when one wants to, say, fill a f64vector with random numbers, unless one has a Sufficiently Smart compiler. Creating a generator by wrapping a procedure that generates one number is easy.~~ The only benefit of a generator interface is when the implementation generates a block of random objects and wishes to buffer it.

I suppose that's fine. I took a quick look at the implementations, and I don't see any that look stateful. I don't understand your reasoning about cost, though: allocating a closure should be cheap compared to filling a vector unless it is small.

I don't think there is anything in 194 that isn't fairly vanilla. They are all standard members of a random number toolkit, except for the u8 through s128 set, whuch are programming-specific.

I suppose that's fine. I took a quick look at the implementations, and I don't see any that look stateful. I don't understand your reasoning about cost, though: allocating a closure should be cheap compared to filling a vector unless it is small. I don't think there is anything in 194 that isn't fairly vanilla. They are all standard members of a random number toolkit, except for the u8 through s128 set, whuch are programming-specific.
Member
Copy link

@johnwcowan wrote in #71 (comment):

I don't understand your reasoning about cost, though: allocating a closure should be cheap compared to filling a vector unless it is small.

Yes, disregard that. I mixed up two lines of thought into something that doesn't make sense.

@johnwcowan wrote in https://codeberg.org/scheme/r7rs/issues/71#issuecomment-18910679: > I don't understand your reasoning about cost, though: allocating a closure should be cheap compared to filling a vector unless it is small. Yes, disregard that. I mixed up two lines of thought into something that doesn't make sense.
Member
Copy link

@phm wrote in #71 (comment):

Personally I would rather an API that returns a single randomly selected object vs. returning a generator. For example (random-normal port mean deviation) instead of (make-normal-generator port mean deviation).

I agree. The random-port-based sketch I'm working on provides procedures like (read-random-u64), which read either from a port argument or the value of the current-random-port parameter.

@johnwcowan wrote in #71 (comment):

I don't think there is anything in 194 that isn't fairly vanilla.

Take a look at the "nonuniform distributions" section of 194. I remember a great deal of debate about the correctness of those implementations, and I'm not keen to repeat it.

@phm wrote in https://codeberg.org/scheme/r7rs/issues/71#issuecomment-18909500: > Personally I would rather an API that returns a single randomly selected object vs. returning a generator. For example `(random-normal port mean deviation)` instead of `(make-normal-generator port mean deviation)`. I agree. The random-port-based sketch I'm working on provides procedures like `(read-random-u64)`, which read either from a port argument or the value of the `current-random-port` parameter. @johnwcowan wrote in https://codeberg.org/scheme/r7rs/issues/71#issuecomment-18910679: > I don't think there is anything in 194 that isn't fairly vanilla. Take a look at the "nonuniform distributions" section of 194. I remember a great deal of debate about the correctness of those implementations, and I'm not keen to repeat it.
Sign in to join this conversation.
No Branch/Tag specified
main
pages
proc-draft
c-trichotomy
global-id-prop-ref
allow-van-tonder-like-expanders
No results found.
Labels
Clear labels
Batteries
Issues for the Batteries volume of the report, providing a standard library
Cleanup
Issues where inconsistencies or other infelicities have arisen
Consent Docket
Items which will be automatically be given a certain resolution if no objection is made by a certain date
Environments
Issues for the Environments volume of the report, providing OS interfaces and similar
Foundations
Issues for the Foundations volume of the report, related to the core language semantics
Macrological Fascicle
Meta
Procedural and organisational issues
Multiple proposals exist
Multiple competing, concrete proposals to resolve this issue exist (whether they are SRFI or still only pre-SRFI)
Pie-in-the-sky ideas
Ideas currently lacking any form of worked-out proposal
Problems without solutions
Problems we *must* solve but haven’t worked out any solution for yet
Procedural Fascicle
Public Comment
Publications
Issues related to the practicalities of making the report available
Question
For information and clarification
R6RS Compatibility
Issue touches on questions of R6RS compatibility
R7RS-small
Issues with the Small language Report
Resolution in draft
A resolution (not necessarily final) is specified in the current draft of the relevant fascicle
Specification exists
Proposals for which a viable, but incomplete specification text exists
SRFI/RnRS spec exists
A SRFI for this proposal exists, or a solution from another RnRS
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
8 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
scheme/r7rs#71
Reference in a new issue
scheme/r7rs
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?