Make API backend pluggable to allow for non-OpenAI models #18

jamesturk commented 2023年03月20日 22:21:19 +01:00 (Migrated from github.com)

This seems like it'll be the most important task to make this more viable for people.

Alternative models will be cheaper, potentially much faster, allow running on someone's own hardware (LLaMa), and allow for more experimentation (e.g. models that are trained on HTML->JSON).

Quite a few models are attention free, which would remove the token limit altogether.

Models

OpenAssistant

No API as of June 2023, their FAQ makes it sound unlikely.

Cohere

TBD, commenter below says it didn't work well, haven't evaluated.

Anthropic Claude

100k limits added in May, as soon as I get access this will be my guinea pig to add support for pluggable models.

Others

Please add comments below if you've tried this approach with others that have an API.

This seems like it'll be the most important task to make this more viable for people. Alternative models will be cheaper, potentially much faster, allow running on someone's own hardware (LLaMa), and allow for more experimentation (e.g. models that are trained on HTML->JSON). Quite a few models are attention free, which would remove the token limit altogether. ## Models ### OpenAssistant No API as of June 2023, their FAQ makes it sound unlikely. ### Cohere TBD, commenter below says it didn't work well, haven't evaluated. ### Anthropic Claude 100k limits added in May, as soon as I get access this will be my guinea pig to add support for pluggable models. ### Others Please add comments below if you've tried this approach with others that have an API.

irux commented 2023年03月27日 00:34:44 +02:00 (Migrated from github.com)

I am actually curious if this would work with other kind of models. I always had the idea to try to use Bert for this kind of things, but I think a following instructions model would be needed for a good performance.

I am actually curious if this would work with other kind of models. I always had the idea to try to use Bert for this kind of things, but I think a following instructions model would be needed for a good performance.

clarkmcc commented 2023年03月27日 05:50:21 +02:00 (Migrated from github.com)

Yes, I'd love to see this on the new alpaca models. The major problem that I see (not understanding how this prompts OpenAI under the hood) is that successful prompts are much trickier with models like alpaca and llama.

Yes, I'd love to see this on the new alpaca models. The major problem that I see (not understanding how this prompts OpenAI under the hood) is that successful prompts are much trickier with models like alpaca and llama.

walking-octopus commented 2023年04月04日 11:15:03 +02:00 (Migrated from github.com)

Hmm.... Didn't Cohere make their models free to call (albeit with a rate limit)? This could make using this much much more viable for scraping a few small pages.

EDIT: Their models seem to be too weird for this, I've tried.

Hmm.... Didn't Cohere make their models free to call (albeit with a rate limit)? This could make using this much much more viable for scraping a few small pages. EDIT: Their models seem to be too weird for this, I've tried.

daankortenbach commented 2023年05月10日 11:30:09 +02:00 (Migrated from github.com)

I'd love to see support for OpenAssistent models.

I'd love to see support for OpenAssistent models.

walking-octopus commented 2023年05月10日 15:46:24 +02:00 (Migrated from github.com)

I'd love to see support for OpenAssistent models.

Perhaps someone could train some seq2seq model precisely for this task...

> I'd love to see support for OpenAssistent models. Perhaps someone could train some seq2seq model precisely for this task...

jamesturk commented 2023年05月10日 17:55:14 +02:00 (Migrated from github.com)

If anyone wants to work on this let me know, I'd love to discuss approaches

If anyone wants to work on this let me know, I'd love to discuss approaches

jamesturk commented 2023年06月07日 00:46:36 +02:00 (Migrated from github.com)

The groundwork for this is there after some recent refactors, I am hoping to get access to Claude soon as with its 100k token limit it'd be amazing to see how it performs. Updating the parent issue w/ the status of some other models as well.

The groundwork for this is there after some recent refactors, I am hoping to get access to Claude soon as with its 100k token limit it'd be amazing to see how it performs. Updating the parent issue w/ the status of some other models as well.

cpoptic commented 2023年07月06日 16:54:42 +02:00 (Migrated from github.com)

How about adding Falcon 7B and/or 40B LLM model support?

How about adding Falcon 7B and/or 40B LLM model support?

Sunishchal commented 2023年07月23日 09:27:20 +02:00 (Migrated from github.com)

@jamesturk I'm very interested in support for Claude 100k. Happy to work on a PR for this if you're welcoming contributors.

@jamesturk I'm very interested in support for Claude 100k. Happy to work on a PR for this if you're welcoming contributors.

ishaan-jaff commented 2023年08月03日 00:18:12 +02:00 (Migrated from github.com)

@jamesturk expanded non OpenAI model coverage in this PR #55

@jamesturk expanded non OpenAI model coverage in this PR #55

jamesturk commented 2023年08月03日 00:54:34 +02:00 (Migrated from github.com)

Update to those tracking this:
In general the approach laid out in PR #55 seems like a great way to go, relying on a well maintained library that abstracts away differences between these models saves this library from needing to reinvent that wheel that others have tackled. I'd been toying with a lightweight version of the same, but hadn't done research on what else was out there yet.

As noted on PR #55 I don't think it's quite ready to be merged in yet, there are other parts of the code base that assume OpenAI that I'll want to check on. (I also just heard about litellm half an hour ago & want to do a tiny bit of due diligence before adding the dependency 😄 )

Update to those tracking this: In general the approach laid out in PR #55 seems like a great way to go, relying on a well maintained library that abstracts away differences between these models saves this library from needing to reinvent that wheel that others have tackled. I'd been toying with a lightweight version of the same, but hadn't done research on what else was out there yet. As noted on PR #55 I don't think it's quite ready to be merged in yet, there are other parts of the code base that assume OpenAI that I'll want to check on. (I also just heard about `litellm` half an hour ago & want to do a tiny bit of due diligence before adding the dependency 😄 )

cornpo commented 2023年11月03日 06:15:06 +01:00 (Migrated from github.com)

This works well with oobabooga's OpenAi extension with Mistral7B and Phind34B.

https://github.com/briansunter/logseq-plugin-gpt3-openai

This works well with oobabooga's OpenAi extension with Mistral7B and Phind34B. https://github.com/briansunter/logseq-plugin-gpt3-openai