聊聊Spring AI Alibaba的BilibiliDocumentReader

序

本文主要研究一下Spring AI Alibaba的BilibiliDocumentReader

BilibiliDocumentReader

community/document-readers/spring-ai-alibaba-starter-document-reader-bilibili/src/main/java/com/alibaba/cloud/ai/reader/bilibili/BilibiliDocumentReader.java

public class BilibiliDocumentReader implements DocumentReader {
 private static final Logger logger = LoggerFactory.getLogger(BilibiliDocumentReader.class);
 private static final String API_BASE_URL = "https://api.bilibili.com/x/web-interface/view?bvid=";
 private final String resourcePath;
 private final ObjectMapper objectMapper;
 private static final int MEMORY_SIZE = 5;
 private static final int BYTE_SIZE = 1024;
 private static final int MAX_MEMORY_SIZE = MEMORY_SIZE * BYTE_SIZE * BYTE_SIZE;
 private static final WebClient WEB_CLIENT = WebClient.builder()
 .defaultHeader(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE)
 .codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(MAX_MEMORY_SIZE))
 .build();
 public BilibiliDocumentReader(String resourcePath) {
 Assert.hasText(resourcePath, "Query string must not be empty");
 this.resourcePath = resourcePath;
 this.objectMapper = new ObjectMapper();
 }
 @Override
 public List<Document> get() {
 List<Document> documents = new ArrayList<>();
 try {
 String bvid = extractBvid(resourcePath);
 String videoInfoResponse = fetchVideoInfo(bvid);
 JsonNode videoData = parseJson(videoInfoResponse).path("data");
 String title = videoData.path("title").asText();
 String description = videoData.path("desc").asText();
 Document infoDoc = new Document("Video information", Map.of("title", title, "description", description));
 documents.add(infoDoc);
 String documentContent = fetchAndProcessSubtitles(videoData, title, description);
 documents.add(new Document(documentContent));
 }
 catch (IllegalArgumentException e) {
 logger.error("Invalid input: {}", e.getMessage());
 documents.add(new Document("Error: Invalid input"));
 }
 catch (IOException e) {
 logger.error("Error parsing JSON: {}", e.getMessage(), e);
 documents.add(new Document("Error parsing JSON: " + e.getMessage()));
 }
 catch (Exception e) {
 logger.error("Unexpected error: {}", e.getMessage(), e);
 documents.add(new Document("Unexpected error: " + e.getMessage()));
 }
 return documents;
 }
 private String extractBvid(String resourcePath) {
 return resourcePath.replaceAll(".*(BV\\w+).*", "1ドル");
 }
 private String fetchVideoInfo(String bvid) {
 return WEB_CLIENT.get().uri(API_BASE_URL + bvid).retrieve().bodyToMono(String.class).block();
 }
 private JsonNode parseJson(String jsonResponse) throws IOException {
 return objectMapper.readTree(jsonResponse);
 }
 private String fetchAndProcessSubtitles(JsonNode videoData, String title, String description) throws IOException {
 JsonNode subtitleList = videoData.path("subtitle").path("list");
 if (subtitleList.isArray() && subtitleList.size() > 0) {
 String subtitleUrl = subtitleList.get(0).path("subtitle_url").asText();
 String subtitleResponse = WEB_CLIENT.get().uri(subtitleUrl).retrieve().bodyToMono(String.class).block();
 JsonNode subtitleJson = parseJson(subtitleResponse);
 StringBuilder rawTranscript = new StringBuilder();
 subtitleJson.path("body").forEach(node -> rawTranscript.append(node.path("content").asText()).append(" "));
 return String.format("Video Title: %s, Description: %s\nTranscript: %s", title, description,
 rawTranscript.toString().trim());
 }
 else {
 return String.format("No subtitles found for video: %s. Returning an empty transcript.", resourcePath);
 }
 }
}

BilibiliDocumentReader使用WebClient去请求B站接口,它从url解析bvid,再根据bvid去请求接口,解析json获取title、description,通过fetchAndProcessSubtitles再去请求subtitle_url获取字幕内容作为document的内容

示例

public class BilibiliDocumentReaderTest {
 private static final Logger logger = LoggerFactory.getLogger(BilibiliDocumentReader.class);
 @Test
 void bilibiliDocumentReaderTest() {
 BilibiliDocumentReader bilibiliDocumentReader = new BilibiliDocumentReader(
 "https://www.bilibili.com/video/BV1KMwgeKECx/?t=7&vd_source=3069f51b168ac07a9e3c4ba94ae26af5");
 List<Document> documents = bilibiliDocumentReader.get();
 logger.info("documents: {}", documents);
 }
}

小结

spring-ai-alibaba-starter-document-reader-bilibili提供了BilibiliDocumentReader用于解析B站的视频url,它请求两次接口,一次获取title和description,一次获取字幕。

doc

• java2ai