Quarkus LangChain4J Extension Allows Developers to Integrate LLMs in Their Quarkus Applications

The Quarkus team have begun working on an extension based on the LangChain4J library, the Java re-implementation of the langchain library (initially implemented in Python or JavaScript). This was inspired by the Devoxx Belgium presentation, "Java Meets AI", by Lize Raes, strategic advisor and advocate speaker for LangChain for Java (LangChain4J).

The extension enables developers to integrate Large Language Models (LLM) into their Quarkus applications. The first public version of Quarkus LangChain4J, 0.1, was released mid-November 2023. The extension has quickly evolved, with releases almost weekly, the latest version being 0.5.1.

InfoQ reached out to Max Rydahl Andersen, Quarkus’ project co-lead to share his opinion on the future development of the extension and how suitable for production it is. He stated:

Using it in a real project? Try it, but the API of langchain4j is still changing so we are still keeping it experimental.

We will keep up with langchain4j and keep expanding it. We are especially interested in expanding support for "open" models, especially those that can be run on your Cloud or on-premise infrastructure.

Andersen believes LLMs can be used in a lot of existing and future enterprise projects and he thinks that the new emerging programming model fits nicely into Quarkus’ existing feature set. The extension allows defining LLM integration points declaratively, similar to the Quarkus REST client: an interface is annotated with @RegisterAiService and subsequently, the LLM will be able to be used by injecting the service everywhere in the application. This approach has the advantage of providing:

• Testability, where the service can be mocked via a fake implementation of the interface;
• Observability, where the metrics annotations will allow developers to monitor the methods;
• Resilience, where fault-tolerance annotations will allow developers to handle failures, timeouts, and other transient issues.

@RegisterAiService
public interface TriageService {
    // methods.
}

When working with LLMs like ChatGPT, most of the interaction is done via natural language prompts, while in classical applications, the interaction is done via programming languages. Unlike traditional code, the quarkus-langchain extension retains that way of interaction with LLMs by allowing the developer to define the scope and the task through natural language. The scope of the LLM may be defined via the @SystemMessage(String) annotation and the task through the @UserMessage(String) annotation.

@RegisterAiService
public interface TriageService {
    @SystemMessage("""
        You are working for a bank, processing reviews about
        financial products. Triage reviews into positive and
        negative ones, responding with a JSON document.
        """
    )
    @UserMessage("""
        Your task is to process the review delimited by ---.
        Apply sentiment analysis to the review to determine
        if it is positive or negative, considering various languages.

        For example:
        - `I love your bank, you are the best!` is a 'POSITIVE' review
        - `J'adore votre banque` is a 'POSITIVE' review
        - `I hate your bank, you are the worst!` is a 'NEGATIVE' review

        Respond with a JSON document containing:
        - the 'evaluation' key set to 'POSITIVE' if the review is
        positive, 'NEGATIVE' otherwise
        - the 'message' key set to a message thanking or apologizing
        to the customer. These messages must be polite and match the
        review's language.

        ---
        {review}
        ---
    """)
    TriagedReview triage(String review);
}

As the knowledge of the Large Language Models is limited to their training set, the Quarkus LangChain4j extension provides two mechanisms to extend its knowledge: tools and document stores.

The tools allow the LLM to interact with the parent application. It accomplishes this by executing actions within an application by calling a REST endpoint or executing a database query. The LLM decides the parameters to be used and how to handle the result. To declare a tool, just use the @Tool annotation on a bean method:

@ApplicationScoped
public class CustomerRepository implements PanacheRepository<Customer> {

    @Tool("get the customer name for the given customerId")
    public String getCustomerName(long id) {
        return find("id", id).firstResult().name;
    }
}

Document Stores are Quarkus’ implementation of the Retrieval Augmented Generation (RAG), a mechanism to extend the limited context of LLMs with relevant documents about the topics of interest (user manuals, internal documentation etc.). The access to the information from the documents involves two steps:

The ingestion process - the documents are parsed and their vector representation is computed and then stored in the document repository. Quarkus provides an Ingestor which facilitates the ingestion of information.

 @Inject
    EmbeddingModel embeddingModel;

    public void ingest(List<Document> documents) {
        var ingestor = EmbeddingStoreIngestor.builder()
                .embeddingStore(store)
                .embeddingModel(embeddingModel)
                .documentSplitter(recursive(500, 0))
                .build();
        ingestor.ingest(documents);
    }
}

The RAG process - before calling the LLM, the document store is queried and the context is enriched. The Quarkus approach uses a Retriever.

@ApplicationScoped
public class RetrieverExample implements Retriever<TextSegment> {

    private final EmbeddingStoreRetriever retriever;

    RetrieverExample(RedisEmbeddingStore store, EmbeddingModel model) {
        retriever = EmbeddingStoreRetriever.from(store, model, 20);
    }

    @Override
    public List<TextSegment> findRelevant(String s) {
        return retriever.findRelevant(s);
    }
}

Currently, the extension provides support for Redis Store, Chroma Store, Pinecone Store, PgVector (PostgreSQL) Store, in-process embeddings or the ability to load CSV files and the ability to interact with commercial (like OpenAI) and open-source models (like Hugging Face or Ollama).

Quarkus joined the AI embedding capabilities train, following Spring Frameworks’ footsteps. The implementation, based on LangChain4j, is supported by Dmytro Liubarskyi - the creator of LangChain4j and his team. Given the fast development pace, the team is looking for feedback and ideas to improve these integrations. According to Andersen, the LLM extension complements nicely the other existing integrations: various data ingestion systems can be integrated (for example, through the Apache Camel integration), while Quarkus’ cloud-native DNA will allow easy and efficient deployments.