Arbelle GPT: Expert shade matching chat that transforms user experience

Summary

Client and objective

Client: Arbelle
Industry: Beauty, makeup and cosmetics

Arbelle helps beauty brands innovate with digital products that transform how they attract and engage customers. Makeup is a visual category and relies on trust. Customers want confidence that a recommended shade will match and that a product meets their personal preferences. With advances in AI, brands can now offer transparent guidance, richer engagement, and credible advice before a shopper ever tries a product in store.

Objective: Build a flexible product that any beauty brand can adopt to deliver virtual try-ons and expert guidance through a chat experience powered by computer vision and a large language model.

The challenge

Arbelle has proprietary technology that recommends and lets shoppers interact with makeup. It is designed for branding and online shopping and works across landing pages, product pages, and online stores. The Arbelle team wanted to elevate this experience and give shoppers expert level guidance through natural conversation.

To make that vision real they needed to close two gaps:
1. The recommender considers skin tone and product availability, but it cannot capture verbal preferences like finish, coverage, ingredients, values, or intent. 2. Interactions are limited. Shoppers cannot ask follow up questions or explore scenarios. A conversational interface would enable flexible and intuitive guidance.

Our approach

Arbelle, powered by Visage Technologies, brought a proven foundation shade finder, that gave the solution team a strong computer vision starting point. The goal was to connect that capability to an LLM so the experience felt natural for shoppers and easy for brand teams to configure and manage.

We designed a flow where computer vision handles shade estimation and fit checks, while the LLM manages conversation, clarifies preferences, and explains the why behind each recommendation. The assistant also invites the shopper to try a virtual look so they can see the result.

Solution overview

Arbelle GPT is a web app that guides a shopper through three steps.

1. Understanding the needs. 

The assistant asks about coverage, finish, skin type, sensitivity concerns, vegan preferences, price range, and any brand constraints.

2. Matching the shade. 

The system uses our face analysis and shade estimation to propose one or more matches from the database of products that the client offers.

3. Recommend products. 

The assistant maps the match to specific products in the brand catalog, explains the reasoning, and offers a virtual try on. 

Behind the scenes, the assistant uses structured tools to fetch catalog data, check shades, and generate short explanations that are easy to read on mobile.

In essence, Arbelle GPT offers two ways to get recommendations.

• With an image or the camera. The shade finder analyzes skin tone and sends the result to the LLM. The assistant then recommends the best matching products.
• Without an image. The user takes a short quiz. Based on the answers the LLM evaluates needs and recommends suitable products.

Technical solution

Architecture.
We implemented a Quart web server for fast request handling and streaming responses. The backend integrates our shade estimation models and a product catalog service backed by a database. The frontend is built in ReactJS for a responsive mobile experience.

LLM integration.

▸ Function calling connects the LLM to shade estimation, catalog search, and virtual try on triggers. ▸ Prompt templates keep responses short and on brand. ▸ Summarized conversation memory maintains context while controlling token use. ▸ Exponential backoff and request queuing protect reliability during traffic spikes and API rate limits. ▸ A guardrail layer checks for policy compliance and prevents prompt injection.

Computer vision.

▸Face analysis and tracking support robust shade estimation across lighting and pose. ▸Models were trained and validated with data from real users and feedback from makeup artists. ▸We added a score rating (1 – 5 stars) so that brands can have analytics and make data driven decisions about further or different suggestions.

Security and privacy

We have taken a privacy first approach and no face images are stored by default. Images are completely processed locally within the user’s web browser.

In terms of cybersecurity the database is not publicly exposed and follows the principle of least privilege while inputs to the LLM are sanitized and guarded against prompt attacks.

Outcomes

▸ Arbelle GPT, a working product experience that combines shade matching, chat guidance, and optional virtual try on in one flow. ▸Faster answers through streaming so shoppers see the assistant think and do not wait for long responses. ▸Short, clear messages that increase comprehension on mobile. ▸A configuration model that lets each brand map shades and stock to its own catalog without engineering changes.

What could come next

One of the core strengths of the product is its flexibility. We built a customizable platform, not a fixed product, so we can shape it to each client.

We tailor the app through

• Data integration. Securely connect brand catalogs, inventory, and pricing
• Conversation tone. Tune the LLM to match the brand voice and guidance style
• Modular capabilities. Add new computer vision modules or recommenders as needed
• Commerce connections. Integrate carts and storefronts based on the client stack

This flexibility unlocks many possibilities for future capabilities. Here are just a few examples of what could be next:

• Direct purchase in chat with cart and checkout handoff to the brand store
• An autonomous assistant for reorders and back in stock alerts with human handoff
• Analytics on common questions, top recommended products, low converting items, and catalog gaps
• Expansion to categories like concealer, blush, and lipstick using the same framework

New exciting projects are in the works in Visage Technologies!