Corrupt Filename Test

What do Scheme implementations do when encountering files with invalid names?

Usage

Compile prepare.c. It is a C program that creates a file with possibly invalid text. It parses \x escapes in its first argument to create raw bytes, and creates a file with the text hello, world.

Running make all will create

• \x80\xFE\xFF with the text "hello, world"
• READ\x80ME.md with the text "hello, world"
• ��� (three replacement characters).

Help I created a weird file and now I can't delete it

Try using a graphical file browser (like Dolphin) or a CLI file browser (like GNU mc).

Behavior

There are no filesystem features in the following implementations: Loko, SKINT, TR7.

Summary:

• Invalid Unicode becomes replacement characters: 1
• Invalid Unicode is truncated: 2
• Encoded into a string: 3 (2 are explicitly documented)
• Raises an exception: 3
• Uses a different type: 2

Chez Scheme

Running (directory-path ".") gives

("prepare" "README.md" "���" "Makefile" "���" "prepare.c")

The invalid bytes are replaced with replacement characters.

Chibi Scheme

Strings returned from (directory-path ".") contain raw byte data, and fail when passed to string->list. However, if treated as opaque, they can be used to open files:

(import (chibi filesystem))
(define d (directory-files "."))
(write d)
;; ("prepare" "README.md" "���" "." "Makefile" corrupted "prepare.c" "..")
(define p (open-input-file (list-ref d 5)))
(read-line p)
;; "hello, world"

CHICKEN 6

CHICKEN 6 encodes bad bytes as surrogate characters. All byte streams are valid strings, although they raise errors when passed to string->utf8.

(import (scheme base) (chicken file))
(define d (directory "."))
d ;; ("prepare.c" corrupted "Makefile" "���" "README.md" "prepare")
(map char->integer (string->list (list-ref d 1)))
;; (56448 56574 56575)
(define p (open-input-file (list-ref d 1)))
(read-line p)
;; "hello, world"

Gambit v4.9.5

Gambit will truncate a file with a bad byte at the location of the bad byte. The file is hence not openable.

Gauche 0.9.15

Gauche has "incomplete strings", although those are deprecated. These incomplete strings are used in this version to denote strings that are invalid Unicode.

(import (scheme base) (scheme file) (file util))
(define d (directory-list "."))
d
;; ("." ".." "Makefile" "README.md"
;; #**"\x52;\x45;\x41;\x44;\x80;\x4d;\x45;\x2e;\x6d;\x64;" "prepare" "prepare.c"
;; #**"\x80;\xfe;\xff;" "���")
(define p (open-input-file (list-ref d 4)))
(read-line p)
;; "hello, world"

Guile 3.0.9

The scandir procedure in (ice-9 ftw) raises a decoding error. Manually scanning using opendir and readdir has the same behavior.

MIT-Scheme 11.2

MIT-Scheme has a system of pathnames that could be abstract the string. However, (directory-read ".") fails with a decoding exception.

Mosh Scheme 0.2.9-rc1

Mosh uses \xFFFFFFxx to encode bytes in strings. (This is contrary to the R6RS.) This allows it to open the test file.

Racket

(directory-list ".") returns path elements, and open-input-file accepts these values.

(define d (directory-list "." #:build? #t))
d
;; '(#<path:./Makefile>
;; #<path:./README.md>
;; #<path:./READ?ME.md>
;; #<path:./prepare>
;; #<path:./prepare.c>
;; #<path:./???>
;; #<path:./���>)
(define p (open-input-file (list-ref d 5)))
(read-line p)
;; "hello, world"

Sagittarius 0.9.13

find-files removes the file with no valid bytes, and removes the invalid bytes from the file with one invalid byte in it.

(import (util file))
(define f (find-files "." :recursive #f))
f
;; ("./Makefile" "./README.md" "./README.md" "./prepare" "./prepare.c" "./���")
(string=? (list-ref f 1) (list-ref f 2))
;; #t

STKlos

(directory-files ".") raises a UTF-8 encoding exception.