Introduction

Imagine you’re shopping online and come across a product you love but don’t know its name. Wouldn’t it be amazing to upload a picture and have the app find it for you?

In this article, we’ll show you how to build exactly that: an image-based product search feature using Spring Boot and Google Cloud Vertex AI.

Overview of the Feature

This feature allows users to upload an image and receive a list of products that match it, making the search experience more intuitive and visually driven.

The image-based product search feature leverages Google Cloud Vertex AI to process images and extract relevant keywords. These keywords are then used to search for matching products in the database.

Technology Stack

• Java 21
• Spring boot 3.2.5
• PostgreSQL
• Vertex AI
• ReactJS

We’ll walk through the process of setting up this functionality step-by-step.

Step-by-Step Implementation

1. Create a new project on Google Console

First, we need to create a new project on Google Console for this.

We need to go to https://console.cloud.google.com and create a new account if you already have one. If you have one, sign in to the account.

If you add your bank account, Google Cloud will offer you a free trial.

Once you have created an account or signed in to an already existing account, you can create a new project.

2. Enable Vertex AI Service

On the search bar, we need to find Vertex AI and enable all recommended APIs.

Vertex AI is Google Cloud’s fully managed machine learning (ML) platform designed to simplify the development, deployment, and management of ML models. It allows you to build, train, and deploy ML models at scale by providing tools and services like AutoML, custom model training, hyperparameter tuning, and model monitoring

Gemini 1.5 Flash is part of Google’s Gemini family of models, specifically designed for efficient and high-performance inference in ML applications. Gemini models are a series of advanced AI models developed by Google, often used in natural language processing (NLP), vision tasks, and other AI-powered applications

Note: For other frameworks, you can use Gemini API directly at https://aistudio.google.com/app/prompts/new_chat. Use the structure prompt feature because you can customize your output to match the input so you will get better results.

3. Create a new prompt that matches your application

At this step, we need to customize a prompt that matching with your application.

Vertex AI Studio has provided a lot of sample prompts at Prompt Gallery. We use sample Image text to JSON to extract keywords that are related to the product image.

My application is a CarShop, so I build a prompt like this. My expectation the model will respond to me with a list of keywords relating to the image.

My prompt: Extract the name car to a list keyword and output them in JSON. If you don’t find any information about the car, please output the list empty.\nExample response: [\”rolls\”, \”royce\”, \”wraith\”]

After we customize a suitable prompt with your application. Now, we go to explore how to integrate with Spring Boot Application.

4. Integrate with Spring Boot Application

I have built an E-commerce application about cars. So I want to find cars by the image.

First, in the pom.xml file, you should update your dependency:

    5.1.2
    26.32.0




  
      
          com.google.cloud
          spring-cloud-gcp-dependencies
          ${spring-cloud-gcp.version}
          pom
          import
      

      
          com.google.cloud
          libraries-bom
          ${google-cloud-bom.version}
          pom
          import
      
  




  
      com.google.cloud
      google-cloud-vertexai
  

After you have done the config in the pom.xml file, you create a config class GeminiConfig.java

• MODEL_NAME: “gemini-1.5-flash”
• LOCATION: “Your location when setting up the project”
• PROJECT_ID: “your project ID ”

import com.google.cloud.vertexai.VertexAI;
import com.google.cloud.vertexai.generativeai.GenerativeModel;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration(proxyBeanMethods = false)
public class GeminiConfig {

    private static final String MODEL_NAME = "gemini-1.5-flash";
    private static final String LOCATION = "asia-southeast1";
    private static final String PROJECT_ID = "yasmini";

    @Bean
    public VertexAI vertexAI() {
        return new VertexAI(PROJECT_ID, LOCATION);
    }

    @Bean
    public GenerativeModel getModel(VertexAI vertexAI) {
        return new GenerativeModel(MODEL_NAME, vertexAI);
    }
}

Second, create layers Service, Controller to implement the find car function. Create class service.

Because the Gemini API responds with markdown format, we need to create a function to help convert to JSON, and from JSON we will convert to List string in Java.

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.cloud.vertexai.api.Content;
import com.google.cloud.vertexai.api.GenerateContentResponse;
import com.google.cloud.vertexai.api.Part;
import com.google.cloud.vertexai.generativeai.*;
import com.learning.yasminishop.common.entity.Product;
import com.learning.yasminishop.common.exception.AppException;
import com.learning.yasminishop.common.exception.ErrorCode;
import com.learning.yasminishop.product.ProductRepository;
import com.learning.yasminishop.product.dto.response.ProductResponse;
import com.learning.yasminishop.product.mapper.ProductMapper;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.multipart.MultipartFile;

import java.util.HashSet;
import java.util.List;
import java.util.Objects;
import java.util.Set;

@Service
@RequiredArgsConstructor
@Slf4j
@Transactional(readOnly = true)
public class YasMiniAIService {

    private final GenerativeModel generativeModel;
    private final ProductRepository productRepository;

    private final ProductMapper productMapper;


    public List findCarByImage(MultipartFile file){
        try {
            var prompt = "Extract the name car to a list keyword and output them in JSON. If you don't find any information about the car, please output the list empty.\nExample response: [\"rolls\", \"royce\", \"wraith\"]";
            var content = this.generativeModel.generateContent(
                    ContentMaker.fromMultiModalData(
                            PartMaker.fromMimeTypeAndData(Objects.requireNonNull(file.getContentType()), file.getBytes()),
                            prompt
                    )
            );

            String jsonContent = ResponseHandler.getText(content);
            log.info("Extracted keywords from image: {}", jsonContent);
            List keywords = convertJsonToList(jsonContent).stream()
                    .map(String::toLowerCase)
                    .toList();

            Set results = new HashSet();
            for (String keyword : keywords) {
                List products = productRepository.searchByKeyword(keyword);
                results.addAll(products);
            }

            return results.stream()
                    .map(productMapper::toProductResponse)
                    .toList();

        } catch (Exception e) {
            log.error("Error finding car by image", e);
            return List.of();
        }
    }

    private List convertJsonToList(String markdown) throws JsonProcessingException {
        ObjectMapper objectMapper = new ObjectMapper();
        String parseJson = markdown;
        if(markdown.contains("```

json")){
            parseJson = extractJsonFromMarkdown(markdown);
        }
        return objectMapper.readValue(parseJson, List.class);
    }

    private String extractJsonFromMarkdown(String markdown) {
        return markdown.replace("

```json\n", "").replace("\n```

", "");
    }
}

We need to create a controller class to make an endpoint for front end

import com.learning.yasminishop.product.dto.response.ProductResponse;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;

import java.util.List;

@RestController
@RequestMapping("/ai")
@RequiredArgsConstructor
@Slf4j
public class YasMiniAIController {

    private final YasMiniAIService yasMiniAIService;


    @PostMapping
    public List findCar(@RequestParam("file") MultipartFile file) {

        var response = yasMiniAIService.findCarByImage(file);
        return response;
    }
}

5. IMPORTANT step: Login to Google Cloud with Google Cloud CLI

The Spring Boot Application can not verify who you are and isn't able for you to accept the resource in Google Cloud.

So we need to log in to Google and provide authorization.

5.1 First we need to install GCloud CLI on your machine

Link tutorial: https://cloud.google.com/sdk/docs/install
Check the above link and install it on your machine

5.2 Login

1. Open your terminal at the project (you must cd into the project)
2. Type: gcloud auth login
3. Enter, and you will see the windows that allow you to login

gcloud auth login

Note: After you log in, credentials are saved in the Google Maven package, and you don’t need to log in again when restart the Spring Boot application.

Conclusion

So these implement above based on my project E-commerce, you can modify matching with your project, and your framework. In other frameworks, not spring boot (NestJs, ..), you can use https://aistudio.google.com/app/prompts/new_chat. and don’t need to create a new Google Cloud account.

You can check the detailed implementation at my repo:

Backend: https://github.com/duongminhhieu/YasMiniShop
Front-end: https://github.com/duongminhhieu/YasMini-Frontend

Happy learning !!!