Simple encryption for people on a byte budget

Question

Overview

Your goal is to implement RC4 encryption. RC4 encryption, invented by Ron Rivest (of RSA fame), was designed to be secure, yet simple enough to be implemented from memory by military soldiers on the battlefield. Today, there are several attacks on RC4, but it's still used in many places today.

Your program should accept a single string with both a key and some data. It will be presented in this format.

\x0Dthis is a keythis is some data to encrypt

The first byte represents how long the key is. It can be assumed the key will no longer than 255 bytes, and no shorter than 1 byte. The data can be arbitrarily long.

Your program should process the key, then return the encrypted data. RC4 encryption and decryption are identical, so using the same key to "encrypt" the ciphertext should return the original plaintext.

How RC4 Works

Initialization

Initialization of RC4 is quite simple. A state array of 256 bytes is initialized to all bytes from 0 to 255.

S = [0, 1, 2, 3, ..., 253, 254, 255]

Key processing

Values in the state are swapped around based on the key.

j = 0
for i from 0 to 255
 j = (j + S[i] + key[i mod keylength]) mod 256
 swap S[i] and S[j]

Encryption

Encryption is accomplished by using the state to generate pseudo-random bytes, which are then XOR'd to the data. Values in the state are swapped around constantly.

i = j = 0
for each byte B in data
 i = (i + 1) mod 256
 j = (j + S[i]) mod 256
 swap S[i] and S[j]
 K = S[(S[i] + S[j]) mod 256]
 output K XOR B

Expected inputs and outputs

Non-printable characters will be shown in \xAB format.

Input: \x01\x00\x00\x00\x00\x00\x00
Output: \xde\x18\x89A\xa3
Output(hex): de188941a3

Input: \x0Dthis is a keythis is some data to encrypt
Output: \xb5\xdb?i\x1f\x92\x96\x96e!\xf3\xae(!\xf3\xeaC\xd4\x9fS\xbd?d\x82\x84{\xcdN
Output(hex): b5db3f691f9296966521f3ae2821f3ea43d49f53bd3f6482847bcd4e

Input: \x0dthis is a key\xb5\xdb?i\x1f\x92\x96\x96e!\xf3\xae(!\xf3\xeaC\xd4\x9fS\xbd?d\x82\x84{\xcdN
Input(hex): 0d746869732069732061206b6579b5db3f691f9296966521f3ae2821f3ea43d49f53bd3f6482847bcd4e
Output: this is some data to encrypt

Input: Sthis is a rather long key because the value of S is 83 so the key length must matchand this is the data to be encrypted
Output: \x96\x1f,\x8f\xa3%\x9b\xa3f[mk\xdf\xbc\xac\x8b\x8e\xfa\xfe\x96B=!\xfc;\x13`c\x16q\x04\x11\xd8\x86\xee\x07
Output(hex): 961f2c8fa3259ba3665b6d6bdfbcac8b8efafe96423d21fc3b13606316710411d886ee07

• thaliaarchi: Which mode of output do you want, or can we choose which? Hex would be preferred
• Daffy: @andrewarchi The output is supposed to be in bytes so it can be cycled back through for decryption.
• Arnauld: Can we also take input as an array of bytes?
• Daffy: @andrewarchi Those notations were just for readability. If your program is supposed to output \xde, then it should be 1 byte long, and converting it to a number (through python's ord() or javascript's .charCodeAt(0)) should return 222 (0xDE).
• Daffy: @Arnauld Considering the two are basically indistinguishable in a lot of languages, yes.

Answer 1

138 bytes, machine code (16-bit x86)

000000 88 9f 00 02 fe c3 75 f8 b9 01 00 89 fa b4 3f cd
000010 21 86 0d ba 00 03 b4 3f cd 21 89 dd 01 f6 02 9e
000020 00 03 8a 04 00 c3 86 87 00 02 88 04 46 45 39 cd
000030 75 02 31 ed 81 fe 00 03 75 e4 81 ee 00 01 31 db
000040 31 d2 ff c6 81 fe 00 03 75 04 81 ee 00 01 8a 04
000050 00 c3 88 c2 86 87 00 02 88 04 00 c2 89 d7 81 c7
000060 00 02 8a 15 89 d7 89 dd 31 db ba 00 03 b9 01 00
000070 b8 00 3f cd 21 87 fa 30 15 85 c0 74 0b 87 fa 43
000080 b4 40 cd 21 89 eb eb b8 cd 20

Running: save to codegolf.com, dosbox:

codegolf.com < input.bin

successfull attempt

I don't know if this gonna count as an entry, but I've decided to do it manually using hex editors. No compilers were used to do this.

ht editor actually has assembler, but honestly I did not know about this until I was finished ̄\(ツ)/ ̄

Why & how

Why: mostly cause I wanted to check if I'm able to do this.

How: I've started with creating byte filled with NOPs and following with a simple part: trying to write first loop that fills State with 0..255 values. I switched to python and quickly wrote python version, just to have something to test against. Next I was simplifying python code into pseudo code / pseudo assembly. Then I I was trying to write small pieces. I decided it'll be easiest to read from stdin, so I started with something small that will read single byte, then I added password reading and key initialization. Figuring out what registers to pick took me some time.

I though adding de/encryption loop will be easy, but first I actually got single byte decode and added whole loop afterwards.

Last step was getting rid of additional nops that I've left between instructions when writing it (ofc that required fixing jumps as well).

You can see small gallery that I tried to make while progressing here.

Dissection

The program relies on some initial values after startup (see resources below).

00000000 889f0002 mov [bx+0200], bl
00000004 fec3 inc bl
00000006 75f8 jnz 0x0

fill in State (at 0x200)

00000008 b90100 mov cx, 0x1
0000000b 89fa mov dx, di
0000000d b43f mov ah, 0x3f
0000000f cd21 int 0x21
00000011 860d xchg [di], cl
00000013 ba0003 mov dx, 0300
00000016 b43f mov ah, 0x3f
00000018 cd21 int 0x21

read length, read password, store password at ds:0x300

0000001a 89dd mov bp, bx
0000001c 01f6 add si, si
0000001e 029e0003 add bl, [bp+0300]
00000022 8a04 mov al, [si]
00000024 00c3 add bl, al
00000026 86870002 xchg [bx+0200], al
0000002a 8804 mov [si], al
0000002c 46 inc si
0000002d 45 inc bp
0000002e 39cd cmp bp, cx
00000030 7502 jnz 0x34
00000032 31ed xor bp, bp
00000034 81fe0003 cmp si, 0300
00000038 75e4 jnz 0x1e

initialize State with a key (BP is used to traverse key, SI is used to traverse State)

0000003a 81ee0001 sub si, 0100
0000003e 31db xor bx, bx
00000040 31d2 xor dx, dx
00000042 ffc6 inc si
00000044 81fe0003 cmp si, 0300
00000048 7504 jnz 0x4e
0000004a 81ee0001 sub si, 0100
0000004e 8a04 mov al, [si]
00000050 00c3 add bl, al
00000052 88c2 mov dl, al
00000054 86870002 xchg [bx+0200], al
00000058 8804 mov [si], al
0000005a 00c2 add dl, al
0000005c 89d7 mov di, dx
0000005e 81c70002 add di, 0200
00000062 8a15 mov dl, [di]

generate pseudo random value (in DL, DH is 0 thx to xor at 0x140)

00000064 89d7 mov di, dx 
00000066 89dd mov bp, bx 
00000068 31db xor bx, bx 
0000006a ba0003 mov dx, 0300 
0000006d b90100 mov cx, 0x1 
00000070 b8003f mov ax, 3f00 
00000073 cd21 int 0x21 
00000075 87fa xchg dx, di 
00000077 3015 xor [di], dl 
00000079 85c0 test ax, ax 
0000007b 740b jz 0x88 
0000007d 87fa xchg dx, di 
0000007f 43 inc bx 
00000080 b440 mov ah, 0x40 
00000082 cd21 int 0x21 
00000084 89eb mov bx, bp 
00000086 ebb8 jmp 0x40 
00000088 cd20 int 0x20 

• store values that we need to preserve (SI - ints won't touch it, BX)
• read char from input, xor it
• quit if end of stream
• output decoded char
• restore values
• loop to 0x40 (reuse xor on DX)

P.S. This probably could be even shorter, but this took 4 evenings, so not sure if I want to spend another one...

Tools and resources

• ht editor - it has 16-bit disasm
• hexplorer - mostly for it's mouse based copy/paste + insert/overwrite
• dosbox - for debugging
• X86 Opcode and Instruction Reference 1.12 and invaluable 16-bit ModR/M Byte
• DOS INT 21h - DOS Function Codes
• 80386 Instruction Set note this is 386, but that's more than enough
• Register values at DOS .COM startup

Answer 2

C (gcc), (削除) 193 188 182 178 171 (削除ここまで) 172 bytes

f(x,l)int*x;{unsigned char*X=x,i=0,j=0,S[256],t;for(;S[i]=++i;);for(;t=S[i],S[i]=S[j+=t+X[1+i%*X]],S[j]=t,t=++i;);for(X+=*X;l--;S[i]-=S[t]=j)*++X^=S[S[i]+=S[t+=j=S[++i]]];}

Try it online!

Edit: Now works with keys longer than 127 bytes.

Edit2: Added testcase with 129 byte key to TIO link.

Slightly less golfed version

f(x,l)int*x;{
 unsigned char*X=x,i=0,j=0,S[256],t;
 // initialize state
 for(;S[i]=++i;);
 // key processing
 for(;t=S[i],S[i]=S[j+=t+X[1+i%*X]],S[j]=t,t=++i;);
 // encrypt
 for(X+=*X;l--;S[i]-=S[t]=j)
 *++X^=S[S[i]+=S[t+=j=S[++i]]];
}

• user58988: You not fear bugs of a general C compiler? It is undefined behaviour s[i]=++i? Yes i know possibly it is only important the compiler one use...
• user58988: For what it is written in "Key processing" part key has to be <= 256 bytes... (because other characters not influence computation and swaps)
• user58988: And why it would not be ok using char* instead of int* in the argument (yes only less characters)? My seems ok even for 129 bytes length key... Undefined behaviour it seems

Answer 3

CPU x86 instruction set, 133 bytes

000009F8 53 push ebx
000009F9 56 push esi
000009FA 57 push edi
000009FB 55 push ebp
000009FC 55 push ebp
000009FD BF00010000 mov edi,0x100
00000A02 29FC sub esp,edi
00000A04 8B6C3C18 mov ebp,[esp+edi+0x18]
00000A08 31DB xor ebx,ebx
00000A0A 8A5D00 mov bl,[ebp+0x0]
00000A0D 45 inc ebp
00000A0E 31C0 xor eax,eax
00000A10 880404 mov [esp+eax],al
00000A13 40 inc eax
00000A14 39F8 cmp eax,edi
00000A16 72F8 jc 0xa10
00000A18 31F6 xor esi,esi
00000A1A 31C9 xor ecx,ecx
00000A1C 89F0 mov eax,esi
00000A1E 31D2 xor edx,edx
00000A20 F7F3 div ebx
00000A22 8A0434 mov al,[esp+esi]
00000A25 02441500 add al,[ebp+edx+0x0]
00000A29 00C1 add cl,al
00000A2B 8A0434 mov al,[esp+esi]
00000A2E 8A140C mov dl,[esp+ecx]
00000A31 88040C mov [esp+ecx],al
00000A34 881434 mov [esp+esi],dl
00000A37 46 inc esi
00000A38 39FE cmp esi,edi
00000A3A 72E0 jc 0xa1c
00000A3C 8B443C1C mov eax,[esp+edi+0x1c]
00000A40 01E8 add eax,ebp
00000A42 722F jc 0xa73
00000A44 48 dec eax
00000A45 89C6 mov esi,eax
00000A47 01DD add ebp,ebx
00000A49 31C0 xor eax,eax
00000A4B 31D2 xor edx,edx
00000A4D 31C9 xor ecx,ecx
00000A4F 39F5 cmp ebp,esi
00000A51 7320 jnc 0xa73
00000A53 FEC2 inc dl
00000A55 8A0414 mov al,[esp+edx]
00000A58 00C1 add cl,al
00000A5A 8A1C0C mov bl,[esp+ecx]
00000A5D 88040C mov [esp+ecx],al
00000A60 881C14 mov [esp+edx],bl
00000A63 00D8 add al,bl
00000A65 8A1C04 mov bl,[esp+eax]
00000A68 8A4500 mov al,[ebp+0x0]
00000A6B 30D8 xor al,bl
00000A6D 884500 mov [ebp+0x0],al
00000A70 45 inc ebp
00000A71 EBDC jmp short 0xa4f
00000A73 01FC add esp,edi
00000A75 5D pop ebp
00000A76 5D pop ebp
00000A77 5F pop edi
00000A78 5E pop esi
00000A79 5B pop ebx
00000A7A C20800 ret 0x8
00000A7D

A7D-9F8=85h=133 bytes but i don't know if calculation is ok because the preciding number of bytes of the same function result 130 bytes... The first argumenti of the function that I name "cript" is the string, the second argumenti is the string lenght (first byte+ key lenght + message lenght). Below there is the assembly language file for obtain that cript routines:

; nasmw -fobj this.asm
section _DATA use32 public class=DATA
global cript
section _TEXT use32 public class=CODE
cript: 
 push ebx
 push esi
 push edi
 push ebp
 push ebp
 mov edi, 256
 sub esp, edi
 mov ebp, dword[esp+ edi+24]
 xor ebx, ebx
 mov bl, [ebp]
 inc ebp
 xor eax, eax
.1: mov [esp+eax], al
 inc eax
 cmp eax, edi
 jb .1
 xor esi, esi
 xor ecx, ecx
.2: mov eax, esi
 xor edx, edx
 div ebx
 mov al, [esp+esi]
 add al, [ebp+edx]
 add cl, al
 mov al, [esp+esi]
 mov dl, [esp+ecx]
 mov [esp+ecx], al
 mov [esp+esi], dl
 inc esi
 cmp esi, edi
 jb .2
 mov eax, dword[esp+ edi+28]
 add eax, ebp
 jc .z
 dec eax
 mov esi, eax
 add ebp, ebx
 xor eax, eax
 xor edx, edx
 xor ecx, ecx
.3: cmp ebp, esi
 jae .z
 inc dl
 mov al, [esp+edx]
 add cl, al
 mov bl, [esp+ecx]
 mov [esp+ecx], al
 mov [esp+edx], bl ; swap S[c] S[r]
 add al, bl
 mov bl, [esp+eax]
 mov al, [ebp]
 xor al, bl
 mov [ebp], al
 inc ebp
 jmp short .3
.z: 
 add esp, edi
 pop ebp
 pop ebp
 pop edi
 pop esi
 pop ebx
 ret 8

below the C file for check results:

// Nasmw -fobj fileasm.asm
// bcc32 -v filec.c fileasm.obj
#include <stdio.h>
void _stdcall cript(char*,unsigned);
char es1[]="\x01\x00\x00\x00\x00\x00\x00";
char es2[]="\x0Dthis is a keythis is some data to encrypt";
char es3[]="\x0dthis is a key\xb5\xdb?i\x1f\x92\x96226円e!\xf3\xae(!\xf3\xea\x43\xd4\x9fS\xbd?d\x82\x84{\xcdN";
char es4[]="Sthis is a rather long key because the value of S is 83 so the key length must matchand this is the data to be encrypted";
void printMSGKeyC(unsigned char* a, unsigned len)
{unsigned i,j,k;
 unsigned char *p,*end;
 printf("keylen = %u\nKey = [", (unsigned)*a);
 for(i=1, j=*a;i<=j;++i) printf("%c", a[i]);
 printf("]\nMessage= [");
 for(p=a+i,end=a+len-1;p<end;++p)printf("%c", *p);
 printf("]\n");
}
void printMSGKeyHex(unsigned char* a, unsigned len)
{unsigned i,j,k;
 unsigned char *p,*end;
 printf("keylen = %u\nKey = [", (unsigned)*a);
 for(i=1, j=*a;i<=j;++i) printf("%02x", a[i]);
 printf("]\nMessage= [");
 for(p=a+i,end=a+len-1;p<end;++p)printf("%02x", *p);
 printf("]\n");
}
main()
{printf("sizeof \"%s\"= %u [so the last byte 0 is in the count]\n", "this", sizeof "this");
 printf("Input:\n");
 printMSGKeyHex(es1, sizeof es1);
 cript(es1, (sizeof es1)-1);
 printf("Afther I cript:\n");
 printMSGKeyHex(es1, sizeof es1);
 printf("Input:\n");
 printMSGKeyC(es2, sizeof es2);
 printMSGKeyHex(es2, sizeof es2);
 cript(es2, (sizeof es2)-1);
 printf("Afther I cript:\n");
 printMSGKeyC(es2, sizeof es2);
 printMSGKeyHex(es2, sizeof es2);
 cript(es2, (sizeof es2)-1);
 printf("Afther II cript:\n");
 printMSGKeyC(es2, sizeof es2);
 printMSGKeyHex(es2, sizeof es2);
 printf("----------------------\n");
 printf("Input:\n");
 printMSGKeyHex(es3, sizeof es3);
 cript(es3, (sizeof es3)-1);
 printf("Afther I cript:\n");
 printMSGKeyHex(es3, sizeof es3);
 printf("----------------------\n");
 printf("Input:\n");
 printMSGKeyHex(es4, sizeof es4);
 cript(es4, (sizeof es4)-1);
 printf("Afther I cript:\n");
 printMSGKeyHex(es4, sizeof es4);
 cript(es4, (sizeof es4)-1);
 printf("Afther II cript:\n");
 printMSGKeyHex(es4, sizeof es4);
 return 0;
}

the results:

 sizeof "this"= 5 [so the last byte 0 is in the count]
Input:
keylen = 1
Key = [00]
Message= [0000000000]
Afther I cript:
keylen = 1
Key = [00]
Message= [de188941a3]
Input:
keylen = 13
Key = [this is a key]
Message= [this is some data to encrypt]
keylen = 13
Key = [746869732069732061206b6579]
Message= [7468697320697320736f6d65206461746120746f20656e6372797074]
Afther I cript:
keylen = 13
Key = [this is a key]
Message= [Á█?iÆûûe!3⁄4«(!3⁄4ÛCȃS¢?déä{═N]
keylen = 13
Key = [746869732069732061206b6579]
Message= [b5db3f691f9296966521f3ae2821f3ea43d49f53bd3f6482847bcd4e]
Afther II cript:
keylen = 13
Key = [this is a key]
Message= [this is some data to encrypt]
keylen = 13
Key = [746869732069732061206b6579]
Message= [7468697320697320736f6d65206461746120746f20656e6372797074]
----------------------
Input:
keylen = 13
Key = [746869732069732061206b6579]
Message= [b5db3f691f9296966521f3ae2821f3ea43d49f53bd3f6482847bcd4e]
Afther I cript:
keylen = 13
Key = [746869732069732061206b6579]
Message= [7468697320697320736f6d65206461746120746f20656e6372797074]
----------------------
Input:
keylen = 83
Key = [74686973206973206120726174686572206c6f6e67206b65792062656361757365207468652076616c7565206f66205320697320383320736f20746865206b6579206c656e677468206d757374206d61746368]
Message= [616e64207468697320697320746865206461746120746f20626520656e63727970746564]
Afther I cript:
keylen = 83
Key = [74686973206973206120726174686572206c6f6e67206b65792062656361757365207468652076616c7565206f66205320697320383320736f20746865206b6579206c656e677468206d757374206d61746368]
Message= [961f2c8fa3259ba3665b6d6bdfbcac8b8efafe96423d21fc3b13606316710411d886ee07]
Afther II cript:
keylen = 83
Key = [74686973206973206120726174686572206c6f6e67206b65792062656361757365207468652076616c7565206f66205320697320383320736f20746865206b6579206c656e677468206d757374206d61746368]
Message= [616e64207468697320697320746865206461746120746f20626520656e63727970746564]

Answer 4

JavaScript (ES6), (削除) 169 (削除ここまで) 168 bytes

Takes input as an array of bytes. Returns another array of bytes.

([l,...s])=>s.slice(l).map(b=>b^S[(S[S[J=J+(t=S[I=I+1&m])&m,I]=x=S[J],J]=t)+x&m],[...S=[...Array(m=255).keys(),m]].map(i=>S[S[i]=S[j=j+(t=S[i])+s[i%l]&m],j]=t,I=J=j=0))

How?

This is essentially a literal implementation of the spec.

We first split the input array into l (length of key) and s (payload data: key + message). Then, in order of execution:

• We initialize the state array S and define m = 255 which is repeatedly used later as a bit mask.

  S = [...Array(m = 255).keys(), m]
  

• We shuffle the state array. The indices I and J which are initialized here are actually used in the next step.

  [...S].map(i =>
   S[S[i] = S[j = j + (t = S[i]) + s[i % l] & m], j] = t,
   I = J = j = 0
  )
  

• We apply the encryption.

  s.slice(l).map(b =>
   b ^ S[
   (S[S[J = J + (t = S[I = I + 1 & m]) & m, I] = x = S[J], J] = t) +
   x & m
   ]
  )
  

Test cases

let f =
([l,...s])=>s.slice(l).map(b=>b^S[(S[S[J=J+(t=S[I=I+1&m])&m,I]=x=S[J],J]=t)+x&m],[...S=[...Array(m=255).keys(),m]].map(i=>S[S[i]=S[j=j+(t=S[i])+s[i%l]&m],j]=t,I=J=j=0))
let toArray = s => [...s].map(c => c.charCodeAt())
let toHexa = s => [...s].map(c => ('0' + c.toString(16)).slice(-2)).join` `
console.log(toHexa(f(toArray("\x01\x00\x00\x00\x00\x00\x00"))))
console.log(toHexa(f(toArray("\x0Dthis is a keythis is some data to encrypt"))))
console.log(String.fromCharCode(...f(toArray("\x0dthis is a key\xb5\xdb?i\x1f\x92\x96\x96e!\xf3\xae(!\xf3\xeaC\xd4\x9fS\xbd?d\x82\x84{\xcdN"))))
console.log(toHexa(f(toArray("Sthis is a rather long key because the value of S is 83 so the key length must matchand this is the data to be encrypted"))))

Answer 5

JavaScript (ES6), 262 bytes

I considered using only chained functions, but opted to golfify the algorithm given here: https://gist.github.com/farhadi/2185197.

A=>eval(`for(C=A[c='charCodeAt']
(),K=A.slice(1,++C),T=A.slice(C),i=j=k=l=m=y=0,s=[],r=[],a=256;i<a;)s[i]=i++
for(;j<a;){t=s[k=(k+s[j]+K[c](j%K.length))%a]
s[k]=s[j]
s[j++]=t}for(;y<T.length;){r[y]=T[c](y++)^s[((t=s[m=(m+s[l=++l%a])%a])+
(s[m]=s[l]))%a]
s[l]=t}r`)

Less Golfed

A=>{
 C=A.charCodeAt()
 K=A.slice(1,++C)
 T=A.slice(C)
 for(i=j=k=l=m=y=0,s=[],r=[];i<256;)s[i]=i++
 for(;j<256;){
 t=s[k=(k+s[j]+K.charCodeAt(j%K.length))%256];
 s[k]=s[j];
 s[j++]=t;
 }
 for(;y<T.length;){
 t=s[m=(m+s[l=(l+1)%256])%256];
 s[m]=s[l];
 s[l]=t;
 r[y]=T.charCodeAt(y++)^s[(s[l]+s[m])%256];
 }
 return r;
}

let f =
A=>eval(`for(C=A[c='charCodeAt']
(),K=A.slice(1,++C),T=A.slice(C),i=j=k=l=m=y=0,s=[],r=[],a=256;i<a;)s[i]=i++
for(;j<a;){t=s[k=(k+s[j]+K[c](j%K.length))%a]
s[k]=s[j]
s[j++]=t}for(;y<T.length;){r[y]=T[c](y++)^s[((t=s[m=(m+s[l=++l%a])%a])+
(s[m]=s[l]))%a]
s[l]=t}r`)
let test = (input, expected) => {
 let output = f(input);
 let chars = output.map(c=>String.fromCharCode(c)).join``;
 let hex = output.map(c=>('0' + c.toString(16)).slice(-2)).join``;
 let success = chars === expected;
 console.log([success ? 'SUCCESS:' : 'ERROR:', output, chars, hex].join`\n`);
}
test('\x01\x00\x00\x00\x00\x00\x00','\xde\x18\x89A\xa3')
test('\x0Dthis is a keythis is some data to encrypt','\xb5\xdb?i\x1f\x92\x96\x96e!\xf3\xae(!\xf3\xeaC\xd4\x9fS\xbd?d\x82\x84{\xcdN')
test('\x0dthis is a key\xb5\xdb?i\x1f\x92\x96\x96e!\xf3\xae(!\xf3\xeaC\xd4\x9fS\xbd?d\x82\x84{\xcdN','this is some data to encrypt')
test('Sthis is a rather long key because the value of S is 83 so the key length must matchand this is the data to be encrypted',
 '\x96\x1f,\x8f\xa3%\x9b\xa3f[mk\xdf\xbc\xac\x8b\x8e\xfa\xfe\x96B=!\xfc;\x13`c\x16q\x04\x11\xd8\x86\xee\x07')

• Daffy: +1 I always appreciate it when people include explanations for golfed code.

Answer 6

R, 189 bytes

\(s){f=256
S=1:f-1
for(i in 1:f)S[c(i,T)]=S[c(T<-(T-1+S[i]+s[(i-1)%%s[1]+2])%%f+1,i)]
i=j=1
sapply(s[0:-s-1],\(B)bitwXor(S[sum(S[c(j,i)]<<-S[c(i<<-(i)%%f+1,j<<-(j-1+S[i])%%f+1)])%%f+1],B))}

Attempt This Online!

Ungolfed code:

rc4=function(s){
 l=s[1]
 k=s[1:l+1]
 d=s[-(0:l+1)]
 # initialization
 S=0:255
 # key processing
 j=0
 for(i in 0:255){
 j=(j+S[i+1]+k[i%%l+1])%%256
 temp=S[i+1]
 S[i+1]=S[j+1]
 S[j+1]=temp
 }
 #encryption
 i=j=0
 ans={}
 for(B in d){
 i=(i+1)%%256
 j=(j+S[i+1])%%256
 temp=S[i+1]
 S[i+1]=S[j+1]
 S[j+1]=temp
 K=S[(S[i+1]+S[j+1])%%256+1]
 ans=c(ans,bitwXor(K,B))
 }
 ans
}

Answer 7

Python 2, 203 bytes

def f(x):
	def h():S[i],S[j]=S[j],S[i]
	m=256;x=map(ord,x);S=range(m);l=x.pop(0);j=0
	for i in S[:]:j=(j+S[i]+x[i%l])%m;h()
	i=j=0
	for b in x[l:]:i=(i+1)%m;j=(j+S[i])%m;h();yield chr(S[(S[i]+S[j])%m]^b)

Try it online!

f is a generator (iterable) of strings.

Ungolfed:

def f(x):
 def h():
 S[i], S[j] = S[j], S[i] # we have to do this two times
 m = 256 # used often
 x = map(ord, x) # get numbers to do stuff with
 S = range(m) # init State
 l = x.pop(0) # get key length and remove the first byte in one go
 j = 0
 for i in S[:]: # shorter than range(256)
 j = (j + S[i] + x[i%l]) % m
 h()
 i = j = 0
 for b in x[l:]: # data comes after the key
 i = (i+1) % m
 j = (j+S[i]) % m
 h()
 yield chr(S[(S[i]+S[j]) % m] ^ b) # convert to str again

Answer 8

Ruby, 234 bytes

Untested.

->s{l=s[0].ord;k=s[1..l];i=s[l+1..-1];o='';s=[*0..255];i,j,q=0;256.times{|i|j=(j+s[i]+k[i%k.size].ord)%256;s[i],s[j]=s[j],s[i]};i.size.times{|k|i=(i+1)%256;q=(q+s[i])%256;s[i],s[q]=s[q],s[i];b=s[(s[i]+s[q])%256];o<<(b^i[k].ord).chr;o}

Answer 9

C, 181 bytes

f(a,n)char*a;{unsigned char A=*a++,i=-1,j=0,k,s[256];for(;s[i]=i;--i);for(;k=s[i],s[i]=s[j+=k+a[i%A]],s[j]=k,k=++i;);for(n-=A,a+=A;--n;*a++^=s[j+=A])j=s[++i],A=s[i]=s[k+=j],s[k]=j;}

thanks to ceilingcat for some bytes less:

f(a,n)char*a;
{unsigned char A=*a++,i=-1,j=0,k,s[256];
 for(;s[i]=i;--i);
 for(;k=s[i],s[i]=s[j+=k+a[i%A]],s[j]=k,k=++i;);
 for(n-=A,a+=A;--n;*a++^=s[j+=A])j=s[++i],A=s[i]=s[k+=j],s[k]=j;
}

f(a,n) in "a" there would be the array of chars 1Byte len + key + message; in n there is the size of the all array of "a" not count the last '0円'. code for test and result would be as the one used for assembly function.

• user58988: @ceilingcat my argument of the function n, is here the length of all the array of characters (1byte+ key + message)... length of the message it has to be: n-k-1 where k is the length of the key and n is the length of all the array of characters .
• user58988: @ceilingcat for me the call " f(a,sizeof a);" is wrong because sizeof calculate as length of 'a' the appended char '0円' at end of the string... So it has to be f(a,(sizeof a) -1)
• user58988: I not agree in this ",*++a^=s[j+=s[i]])" it seems to me that here it read 2 times to s array, and in one another similar instruction here the result was different...

Answer 10

APL(NARS), 329 chars, 658 bytes

xor←{⍺<0: ̄1⋄⍵<0: ̄1⋄⍺=0:⍵⋄⍵=0:⍺⋄k←⌈/{⌊1+2⍟⍵} ̈⍺⍵⋄(2⍴⍨≢m)⊥m←↑≠/{(k⍴2)⊤⍵} ̈⍺⍵}
∇r←C w;s;i;j;l;t;b;x
k←256⋄s← ̄1+⍳k⋄i←j←0⋄l← ̄1+⎕AV⍳↑w
t←s[1+i]⋄j←k∣ ̄1+j+t+⎕AV⍳w[2+l∣i]⋄s[1+i]←s[1+j]⋄s[1+j]←t⋄i×ばつ⍳i<k
i←j←0⋄b←≢w⋄l+←1
l×ばつ⍳l>b⋄i←k∣i+1⋄t←s[1+i]⋄j←k∣j+t⋄s[1+i]←s[1+j]⋄s[1+j]←t
x←s[1+k∣t+s[1+i]] xor ̄1+⎕AV⍳w[l]⋄w[l]←⎕AV[1+x]⋄→4
r←w
∇

as always the error check would be demanded to someone other... This seems to be ok on input and output, test:

 str2Hex←{∊{'0123456789ABCDEF'[1+{(2⍴16)⊤⍵}⍵]} ̈{ ̄1+⎕AV⍳⍵}⍵}
 str2Hex ⎕AV[2,1,1,1,1,1,1]
01000000000000
 str2Hex C ⎕AV[2,1,1,1,1,1,1]
0100DE188941A3
 str2Hex C C ⎕AV[2,1,1,1,1,1,1]
01000000000000
 str2Hex C ⎕AV[14],'this is a keythis is some data to encrypt'
0D746869732069732061206B6579B5DB3F691F9296966521F3AE2821F3EA43D49F53BD3F6482847BCD4E
 C C ⎕AV[14],'this is a keythis is some data to encrypt'
this is a keythis is some data to encrypt
 str2Hex C 'Sthis is a rather long key because the value of S is 83 so the key length must matchand this is the data to be encrypted'
5374686973206973206120726174686572206C6F6E67206B65792062656361757365207468652076616C7565206F662053
 20697320383320736F20746865206B6579206C656E677468206D757374206D61746368961F2C8FA3259BA3665B6D
 6BDFBCAC8B8EFAFE96423D21FC3B13606316710411D886EE07
 C C 'Sthis is a rather long key because the value of S is 83 so the key length must matchand this is the data to be encrypted'
Sthis is a rather long key because the value of S is 83 so the key length must matchand this is th
 e data to be encrypted

Yes all can be reduced...but for example make more little the xor function could mean make it less general...

Answer 11

Rust 348

fn rc4(n:Vec<u8>)->Vec<u8>{let(mut s,mut j,mut t,l)=((0..256).collect::<Vec<usize>>(),0,0,n[0] as usize);let(key,msg)=n[1..].split_at(l);(0..256).fold(0,|a,i|{j=(a+s[i]+key[i%l] as usize)%256;t=s[i];s[i]=s[j];s[j]=t;j});j=0;(1..).zip(msg.iter()).map(|(i,b)|{j=(j+s[i])%256;t=s[i];s[i]=s[j];s[j]=t;s[(s[i]+s[j])%256]as u8^b}).collect::<Vec<u8>>()}

This is quite terribly large, I am hoping maybe someone could provide some suggestions.

ungolfed: on play.rust-lang.org playground

• ceilingcat: Suggest k instead of key and m instead of msg and foo&255 instead of (foo)%256