How to create a Shopify theme based testing

Shopify Theme-Based Split URL Testing allows you to test two or more completely different theme versions of your Shopify store to identify which layout, design, or user flow delivers higher conversions. Unlike traditional A/B testing, which tests changes within a single page, this method lets you run experiments across different themes.

For example, you might want to compare your current theme with a newer one, or test a version of your store with a festive layout against a standard layout. CustomFit.ai splits your visitors between them based on your defined targeting or automatically distributes traffic equally.

This type of testing is especially useful when:

  • You’re redesigning your Shopify store and want to test the new layout without affecting your live site.

  • You want to compare fast-loading, minimal themes vs rich media-heavy themes.

  • You want to validate a theme optimized for mobile before making it live for all users.

Steps to create a Theme based testing

Create an experience

  • Log in to the dashboard and navigate to the Experience page from the sidebar.

  • Click on the 'Create Experience' button.

  • After giving your test a name and a description, click the "Create Experience" button.

Add page paths

  • Click on the 'Add pages' button.

  • From the drop-down, select the All Pages option and click on the 'Add' button.

Create behaviors

  • Click on the 'Add Behavior' button.

  • Then give a proper Name and Description for your behavior.

  • Click on the 'Add' button.

Configure Split URL

  • Click on the 'open visual editor' button present on the behavior.

  • Click on the </> icon present in the toolbar.

  • From the dropdown choose the 'Global JavaScript' option.

  • Copy and paste the below code in the JavaScript editor.

(function() {
  var themeName = 'newTestTheme', themeQueryParam = {
        _ab: 0,
        _fd: 0,
        _sc: 1,
        preview_theme_id: 1234567891
    };
    if(Shopify && Shopify.theme && Shopify.theme.id != themeQueryParam.preview_theme_id) {
        document.body.style.display="none";
          
    // add theme details above
    var updateQueryStringParameter = function(uri, key, value) {
        var re = new RegExp("([?&])" + key + "=.*?(&|#|$)", "i");
        if( value === undefined ) {
            if (uri.match(re)) {return uri.replace(re, '$1$2');} else {return uri;}
        } else {
            if (uri.match(re)) {return uri.replace(re, '$1' + key + "=" + value + '$2');
            } else {
                var hash =  '';
                if( uri.indexOf('#') !== -1 ){hash = uri.replace(/.*#/, '#');uri = uri.replace(/#.*/, '');}
                var separator = uri.indexOf('?') !== -1 ? "&" : "?";
                return uri + separator + key + "=" + value + hash;
            }
        }
    };
    var themeUrl = window.location.href;
    for (var key in themeQueryParam) {
        themeUrl = updateQueryStringParameter(themeUrl, key, themeQueryParam[key]);
    }
  
        window.location.href = themeUrl;
        

    }
})();

const style = document.createElement('style');
style.textContent = `#PBarNextFrameWrapper { display: none !important; }`;
document.head.appendChild(style);

  • In the editor, go to line number 6 and change the preview_theme_id to your new theme id.

  • Also, add the following JS by clicking on JS for Default behavior to hide the preview bar.

// Some codeconst style = document.createElement('style');
style.textContent = `#PBarNextFrameWrapper { display: none !important; }`;
document.head.appendChild(style);

  • Click on the Save Changes button.

  • Click on the 'Finalize & Exit' button.

Targeting audience

You can target a Shopify theme to specific audience segments. For example, you can restrict the Shopify theme to visitors coming from a particular campaign. Alternatively, you can target it at all visitors.

  • Click on the 'Add Rules' button.

  • Choose the audience category you wish to target for the Shopify theme test from the pop-up.

  • Then, from the attribute dropdown, select one of the attributes. For the above example, the audience category will be UTM Parameters, and the attribute will be UTM Campaign.

  • Then, from the operation dropdown, select one of the operations, such as 'is one of' or 'is not one of', based on your requirement. For the above example, the operation type will be 'is one of'.

  • Then, provide the value in the input field. For the above example, the value will be the campaign name. You can enter multiple values. In a few of the cases, you can see the drop-down option under the values section.

  • Finally, under the targeting behavior dropdown, select the percentage rollout option. Then, configure the value as 50% for the default and 50% for the newly created behavior.

  • Once the target audience is configured, make sure to click on the 'Save' button.

Add goals

In the experience goals tab, perform the below steps to add a goal.

  • Click on the 'Add goals' button.

  • Then, give the goal name and select the event type based on your requirement. For example, if you want to track the number of ATC's, then select the Shopify option from the dropdown.

  • Next, from the events list, select the event that matches your requirement. In this case, let's choose the Product Added to cart option from the dropdown.

  • If you want to add a specific product to the track, then click on Add rules, and then enter the specific product ID in the rule.

  • Finally, select the Goal evaluation type from the dropdown based on your requirement. For example, select Unique visitors if you want to see how many unique visitors performed Add to cart for the specified product. Atlast click on the Create button.

By default, the first goal event that you add will be considered the primary goal. If you have multiple goal events, then you can make any one of the goal events the primary goal event.

Taking the test live

To make the Shopify theme test live, click on the power switch toggle.

Important Note

After powering off or terminating a theme-level A/B test, it's crucial to immediately delete the test theme that was created during the experience. This is because Shopify may store theme identifiers in the user's cookies. If the test theme is not deleted, returning visitors might continue to see the test version even after the experience is turned off — leading to inconsistent behavior and inaccurate data.

To avoid this:

  • Delete the test theme associated with the A/B experience right after turning it off.

  • If you want to keep a backup of the theme, duplicate it first, then proceed to delete the original test theme.

  • Make sure the theme you delete is the one used specifically for testing — not your live or main theme.

Following these steps ensures a clean rollback and prevents any residual effects from the A/B test.

PreviousHow to create a Split URL testNextHow to create an AI Conversion Optimize experience

Last updated

Was this helpful?