How To Add Tags From Product Images Using ChatGPT Vision API

Time needed: 10 minutes

To get started, be sure to add MESA on your Shopify store. Then, follow these step-by-step instructions.

1. First, start the workflow with the Shopify > Product Created trigger

   Choose the Shopify app then, select the Product Created step.

2. Next, add a Filter step

   To add a Filter, choose the built-in tool from the action menu.

3. Next, check to make sure the product has images

   Use the variable selector to assign the Shopify > Product > Image > Src value to our Filter’s rule. If prompted, choose just the first item returned.

4. Then, choose “Is Not Empty” as the condition to Filter against

   This will ensure the Filter checks that the product has at least one image to pass through the workflow.

5. Next, Add the Transform built-in tool, Mapping action

   Use the built-in tool, Transform and choose the Transform Mapping action step.

6. Add the Key, Value details for reference later in the workflow

   Define the Key as “Prompt” and the Value as:
   
   Write a comma-separated list of search engine tags for this product:
   {% if shopify.body_html %}
   The description of the product for reference is:
   {{shopify.body_html}}
   {% endif %}
   
   This preparation step sets up the prompt to be served to ChatGPT while given the AI context for the image to describe.

7. Next, add the Code built-in tool

   Use the built-in tool, Code for the next step of this workflow.

8. Replace the JS script with the following snippet

   Select all and replace the existing code snippet with the following. This code will connect to ChatGPT using the GPT-4-TURBO model which accepts the image URLs and text strings.
   
   const Mesa = require('vendor/Mesa.js');

/**
* A Mesa Script exports a class with a script() method.
*/
module.exports = new class {

/**
* Mesa Script
*
* @param {object} payload The payload data
* @param {object} context Additional context about this task
*/
script = async (payload, context) => {
const vars = context.steps;

// Retrieve API key from Mesa credentials
let credential = JSON.parse(Mesa.credential.get('chatgpt'));
Mesa.log.info("credential: ", credential);

// Headers for the request
let options = {
"headers": {
"Content-Type": "application/json",
"Authorization": "Bearer " + credential.apikey,
}
};

// Define the API endpoint
let url = 'https://api.openai.com/v1/chat/completions';

// Prepare the content array (text + image URLs)
let content = [
// Add the text prompt as the first part
{ type: "text", text: vars.transform.prompt }
];

// Loop through each image in the Shopify data and add it as an image_url
for (let image of vars.shopify.images) {
content.push({
type: "image_url",
image_url: {
url: image.src // Image URL string
}
});
}

// Define the payload for the request
let data = {
"model": "gpt-4-turbo", // Adjust model as needed, e.g., "gpt-4o-mini" if that's available for images
"messages": [
{
"role": "user",
"content": content
}
],
"max_tokens": 100,
};

// Make the POST request using Mesa.request.post
let result = await Mesa.request.post(url, JSON.stringify(data), options);

// Handle the result from the OpenAI API
if (result.error) {
throw new Error(result.error.message);
}

// Set the task's external data with the response from OpenAI
Mesa.trigger.setTaskExternalData({
"label": result.choices[0].message.content,
});

// Pass the response to the next workflow step
Mesa.output.next({
"tags": result.choices[0].message.content
});
}
}


9. Next, add the Shopify > Product Add Tag action step

   Choose the Shopify app, then search or select the Product Add Tag step.

10. Next, assign the Product ID and ChatGPT response as the Tag value

    For tags, use the following snippet: {{custom.tags}}
    
    This uses the response from the prior code step which ChatGPT is generating.

11. Lastly, save your changes and test your workflow!

    Save your workflow often to ensure data variables for each step are coming through as expected. Always test your workflow, end-to-end and see each step running properly then, turn your workflow on!