Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Spark NLP 5.5.2: GGUF Embeddings, New HTML/Email/Word Ingestion, Enhanced OpenVINO Support, a New Q&A Annotator, and More Enhancements & Fixes #14487

Discussion options

📢 Spark NLP 5.5.2

We’re thrilled to introduce the latest enhancements and new features in this release of Spark NLP! These additions bring more powerful model inference capabilities, seamless data ingestion methods, and greater flexibility for scaling your NLP workflows.

Upgrade today to take advantage of these new capabilities and improvements. As always, we look forward to your feedback and contributions, and thank you for being part of the Spark NLP community!

🔥 New Features & Enhancements

🚀 Major New Features

  • OpenVINO Support for Transformers (#14408)
    Many popular transformer-based annotators now leverage OpenVINO for faster inference on Intel hardware. Enjoy speedier pipelines across a wide array of models—such as DeBerta, DistilBert, RoBerta, XlmRoBerta, Albert, and more—enabling efficient, production-grade NLP at scale.

  • BLIPForQuestionAnswering Transformer (#14422)
    Introducing BLIPForQuestionAnswering, a new image-based question-answering transformer. Simply provide an image and a question, and BLIP will deliver contextually relevant answers. Perfect for use cases in image analysis, e-commerce, and beyond.

  • AutoGGUFEmbeddings Annotator (#14433)
    Seamlessly integrate AutoGGUFModels into your NLP pipeline. The new AutoGGUFEmbeddings annotator provides dense vector embeddings, making it easier than ever to incorporate advanced sentence embeddings into your workflows. We’ve included an end-to-end notebook to help you get started right away.

📜 New Data Ingestions

  • Parsing HTML to DataFrames (#14449)
    Need to analyze web content at scale? Use sparknlp.read().html() to parse local or remote HTML files into structured Spark DataFrames. This new feature makes web-scale data analysis and downstream NLP tasks more accessible and scalable.

  • Email Content to DataFrames (#14455)
    Leverage sparknlp.read().email() to transform email content into organized DataFrames. Analyze communications, extract insights, and enrich your NLP pipelines with minimal effort. (Requires [SPARKNLP-1092] Adding support to read HTML files #14449 to be merged first.)

  • Microsoft Word Document Parsing (#14476)
    Turn .docx and .doc files into structured Spark DataFrames for streamlined integration into your NLP projects. From enterprise documents to reports, this feature simplifies data preparation and analysis at scale.

🐛 Bug Fixes

  • Microsoft Fabric Integration (#14467)
    We’ve added support for Microsoft Fabric to store and retrieve word embeddings efficiently. Leverage your existing infrastructure to scale Spark NLP solutions more effectively.

  • cuDNN Upgrade Instructions for Databricks (#14451)
    Easily upgrade cuDNN on Databricks to accelerate ONNX model inference on GPU, and take advantage of updated installation instructions for a cleaner setup.

  • Metadata Preservation in ChunkEmbeddings (#14462)
    ChunkEmbeddings now retain original metadata, ensuring richer context and more meaningful insights in your downstream tasks.

  • Default Names and Languages for New Annotators (#14469)
    We’ve standardized default names and languages in our seq2seq annotators for better clarity, consistency, and ease of use.

📦 Dependencies

Updated:

  • Jsoup has been updated from 1.18.1 to 1.18.2 to ensure compatibility and maintain security and performance standards.

New Additions for Email and Document Parsing:

  • Jakarta Mail (jakarta.mail:jakarta.mail-api:2.1.3): Added to support parsing and processing email content.

  • Angus Mail (org.eclipse.angus:angus-mail:2.0.3): Complementary mail handling library integrated for more robust email parsing capabilities.

  • Apache POI (org.apache.poi:poi-ooxml:4.1.2 & org.apache.poi:poi-scratchpad:4.1.2): Introduced for parsing Word documents (.docx and .doc) into structured DataFrames, enabling seamless integration of document-based data into Spark NLP workflows.

📝 Models

We have added more than 50,000 new models and pipelines. The complete list of all 83,000+ models & pipelines in 230+ languages is available on our Models Hub.

❤️ Community support

  • Slack For live discussion with the Spark NLP community and the team
  • GitHub Bug reports, feature requests, and contributions
  • Discussions Engage with other community members, share ideas,
    and show off how you use Spark NLP!
  • Medium Spark NLP articles
  • JohnSnowLabs official Medium
  • YouTube Spark NLP video tutorials

Installation

Python

#PyPI
pip install spark-nlp==5.5.2

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:5.5.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.5.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:5.5.2

Apple Silicon (M1 & M2)

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.5.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-silicon_2.12:5.5.2

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.5.2
pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:5.5.2

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, and 3.4.x:

<dependency>
 <groupId>com.johnsnowlabs.nlp</groupId>
 <artifactId>spark-nlp_2.12</artifactId>
 <version>5.5.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
 <groupId>com.johnsnowlabs.nlp</groupId>
 <artifactId>spark-nlp-gpu_2.12</artifactId>
 <version>5.5.2</version>
</dependency>

spark-nlp-silicon:

<dependency>
 <groupId>com.johnsnowlabs.nlp</groupId>
 <artifactId>spark-nlp-silicon_2.12</artifactId>
 <version>5.5.2</version>
</dependency>

spark-nlp-aarch64:

<dependency>
 <groupId>com.johnsnowlabs.nlp</groupId>
 <artifactId>spark-nlp-aarch64_2.12</artifactId>
 <version>5.5.2</version>
</dependency>

FAT JARs

What's Changed

New Contributors

Full Changelog: 5.5.1...5.5.2

This discussion was created from the release Spark NLP 5.5.2: GGUF Embeddings, New HTML/Email/Word Ingestion, Enhanced OpenVINO Support, a New Q&A Annotator, and More Enhancements & Fixes.
You must be logged in to vote

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
1 participant

AltStyle によって変換されたページ (->オリジナル) /