Enabling SQLite FTS5 module by default. #199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

serpulga wants to merge 1 commit into sql-js:master

from serpulga:enable-fts5

Open

Enabling SQLite FTS5 module by default. #199

serpulga wants to merge 1 commit into sql-js:master from serpulga:enable-fts5

Conversation

serpulga

Copy link

@serpulga serpulga commented May 15, 2017

Compiled using EMSCRIPTEN 1.37.9.

Tests:

$ ./test/run.sh 
Testing ./test/test_blob.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
Passed.
Testing ./test/test_database.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
Passed.
Testing ./test/test_errors.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
Passed.
Testing ./test/test_extension_functions.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
Passed.
Testing ./test/test_functions.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
Passed.
Testing ./test/test_issue128.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
Passed.
Testing ./test/test_issue55.js...	Passed.
Testing ./test/test_issue73.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
Passed.
Testing ./test/test_issue76.js...	Passed.
Testing ./test/test_node_file.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
Passed.
Testing ./test/test_statement.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
Passed.
Testing ./test/test_transactions.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
Passed.
Testing ./test/test_worker.js...	writeStringToMemory is deprecated and should not be called! Use stringToUTF8() instead!
assert.js:90
 throw new assert.AssertionError({
 ^
AssertionError: Reading BLOB
 at Worker.worker.onmessage (/Users/serpulga/Documents/src/sql.js/test/test_worker.js:23:14)
 at ChildProcess.<anonymous> (/Users/serpulga/Documents/src/sql.js/node_modules/workerjs/index.js:25:38)
 at emitTwo (events.js:87:13)
 at ChildProcess.emit (events.js:172:7)
 at handleMessage (internal/child_process.js:695:10)
 at Pipe.channel.onread (internal/child_process.js:440:11)
Fail!\e[0m
Warning\e[0m : 12 tests passed out of 13

Same error I get on master [6dc4ac5].

@serpulga


 Enabling SQLite FTS5 module by default.

7e7ce61

Compiled using EMSCRIPTEN 1.37.9.

@petermolnar

Copy link

petermolnar commented Jan 15, 2018

+1 for this; if the FTS5 engine is on, that'd allow search indexes created by other languages to be used in JS in the browser.

@lovasoa

Copy link

Member

lovasoa commented Jan 15, 2018

sql.js is already quite a huge dependency to have. I don't think adding other modules by default is a good idea.

But compiling different versions for different needs is a good idea.

@3CE8D2BAC65BDD6AA9

Copy link

3CE8D2BAC65BDD6AA9 commented Mar 9, 2018 •

edited

Loading

I believe I am able to have FTS5 running. Here are the procedures:

modify Makefile by adding "-DSQLITE_ENABLE_FTS5" in the CFLAGS line. So, the complete CFLAGS line will become CFLAGS=-O2 -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_DISABLE_LFS -DLONGDOUBLE_TYPE=double -DSQLITE_THREADSAFE=0 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS5
make
A few .js files will be created at the js directory. I tested worker.sql.js only so far
In order to test it, I cloned http://kripken.github.io/sql.js/GUI/ to my server and then delete the line <script src="codemirror/mode/sql/sql.js"></script>
Modify gui.js to point to my own worker.sql.js at this line <a href="https://github.com/kripken/sql.js"> to var worker = new Worker("worker.sql.js");
run the gui page
I typed the commands from http://www.sqlitetutorial.net/sqlite-full-text-search/ into the textarea box. All FTS5 commands work perfectly.

In the next few weeks, I will try to test the performance of FTS5 by adding thousands of documents.

@serpulga

Copy link

Author

serpulga commented Mar 9, 2018

@3CE8D2BAC65BDD6AA9 Thank you for posting the steps. I was also able to run FTS5 but this pull request was to enable it by default inside sql.js; but since I agree with @lovasoa (we shouldn't add modules that most people won't use) I didn't even try to fix the failing unit tests.

brody4hire

brody4hire reviewed

Mar 9, 2018

View reviewed changes

Makefile

@@ -4,7 +4,7 @@ EMSCRIPTEN?=/usr/bin

EMCC=$(EMSCRIPTEN)/emcc

CFLAGS=-DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_DISABLE_LFS -DLONGDOUBLE_TYPE=double -DSQLITE_INT64_TYPE="long long int" -DSQLITE_THREADSAFE=0 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS

CFLAGS=-DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_DISABLE_LFS -DLONGDOUBLE_TYPE=double -DSQLITE_INT64_TYPE="long long int" -DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS5

Copy link

@brody4hire brody4hire Mar 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change the SQLITE_THREADSAFE default setting? Was this intended?

Copy link

@3CE8D2BAC65BDD6AA9 3CE8D2BAC65BDD6AA9 Mar 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed the same thing but no clue why there is such a different.

@kimown kimown mentioned this pull request

Apr 16, 2018

Uncaught Error: no such function: json_extract #237

Closed

@dinedal

Copy link

Collaborator

dinedal commented Aug 2, 2018

I am open to the idea of making such compile time extensions into different versions of sql.js (such as memory-growth, debug, etc).

Would that be a good solution here? Can we expect future extensions to pass the standard test suite? Would a standard procedure for adding such extensions help in the future?

@pamelafox pamelafox mentioned this pull request

Sep 19, 2018

Make more strings translatable Khan/live-editor#685

Merged

@lovasoa lovasoa force-pushed the master branch from b3211a6 to e902ec2 Compare

February 29, 2020 11:04

@lovasoa lovasoa force-pushed the master branch from 351ffdd to 6e7ec33 Compare

October 11, 2020 20:00

@jrocha

Copy link

jrocha commented Oct 4, 2023

Would be nice to have an extra flavor with FTS5 enabled.

@jerefrer

Copy link

jerefrer commented Nov 16, 2023

I also wonder if that would be a good idea to include it by default, but I definitely would love to have a ready-made version with FTS5 enabled.

@arcanis arcanis mentioned this pull request

Sep 18, 2024

Enables FTS5 by default #594

Open

Labels

None yet

8 participants

@serpulga @petermolnar @lovasoa @3CE8D2BAC65BDD6AA9 @dinedal @jrocha @jerefrer @brody4hire

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enabling SQLite FTS5 module by default. #199

Are you sure you want to change the base?

Enabling SQLite FTS5 module by default. #199

Uh oh!

Conversation

@serpulga serpulga commented May 15, 2017

Uh oh!

petermolnar commented Jan 15, 2018

Uh oh!

lovasoa commented Jan 15, 2018

Uh oh!

3CE8D2BAC65BDD6AA9 commented Mar 9, 2018 •

edited

Loading

Uh oh!

Uh oh!

serpulga commented Mar 9, 2018

Uh oh!

@brody4hire brody4hire Mar 9, 2018

Choose a reason for hiding this comment

Uh oh!

@3CE8D2BAC65BDD6AA9 3CE8D2BAC65BDD6AA9 Mar 10, 2018

Choose a reason for hiding this comment

Uh oh!

dinedal commented Aug 2, 2018

Uh oh!

jrocha commented Oct 4, 2023

Uh oh!

jerefrer commented Nov 16, 2023

Uh oh!

Uh oh!

Enabling SQLite FTS5 module by default. #199

Are you sure you want to change the base?

Enabling SQLite FTS5 module by default. #199

Uh oh!

Conversation

@serpulga serpulga commented May 15, 2017

Uh oh!

petermolnar commented Jan 15, 2018

Uh oh!

lovasoa commented Jan 15, 2018

Uh oh!

3CE8D2BAC65BDD6AA9 commented Mar 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serpulga commented Mar 9, 2018

Uh oh!

@brody4hire brody4hire Mar 9, 2018

Choose a reason for hiding this comment

Uh oh!

@3CE8D2BAC65BDD6AA9 3CE8D2BAC65BDD6AA9 Mar 10, 2018

Choose a reason for hiding this comment

Uh oh!

dinedal commented Aug 2, 2018

Uh oh!

jrocha commented Oct 4, 2023

Uh oh!

jerefrer commented Nov 16, 2023

Uh oh!

Uh oh!

3CE8D2BAC65BDD6AA9 commented Mar 9, 2018 •

edited

Loading