How to create a Shopify theme based testing

Shopify Theme-Based Split URL Testing allows you to test two or more completely different theme versions of your Shopify store to identify which layout, design, or user flow delivers higher conversions. Unlike traditional A/B testing, which tests changes within a single page, this method lets you run experiments across separate URLs — each hosting a different Shopify theme or theme variant.

For example, you might want to compare your current theme with a newer one, or test a version of your store with a festive layout against a standard layout. Each theme is hosted on a different URL (e.g., yourstore.com/a vs yourstore.com/b), and CustomFit.ai splits your visitors between them based on your defined targeting or automatically distributes traffic equally.

This type of testing is especially useful when:

• You’re redesigning your Shopify store and want to test the new layout without affecting your live site.

• You want to compare fast-loading, minimal themes vs rich media-heavy themes.

• You want to validate a theme optimized for mobile before making it live for all users.

Steps to create a Theme based testing

Create an experience

• Log in to the dashboard and navigate to the Experience page from the sidebar.

• Click on the 'Create Experience' button.

• After giving your test a name and a description, click the "Create Experience" button.

Add page paths

• Click on the 'Add pages' button.

• From the drop-down, select the All Pages option and click on the 'Add' button.

Create behaviors

• Click on the 'Add Behavior' button.

• Then give a proper Name and Description for your behavior.

• Click on the 'Add' button.

Configure Split URL

• Click on the 'open visual editor' button present on the behavior.

• Click on the </> icon present in the toolbar.

• From the dropdown choose the 'Global JavaScript' option.

• Copy and paste the below code in the JavaScript editor.

(function() {
  var themeName = 'newTestTheme', themeQueryParam = {
        _ab: 0,
        _fd: 0,
        _sc: 1,
        preview_theme_id: 1234567891
    };
    if(Shopify && Shopify.theme && Shopify.theme.id != themeQueryParam.preview_theme_id) {
        document.body.style.display="none";
          
    // add theme details above
    var updateQueryStringParameter = function(uri, key, value) {
        var re = new RegExp("([?&])" + key + "=.*?(&|#|$)", "i");
        if( value === undefined ) {
            if (uri.match(re)) {return uri.replace(re, '$1$2');} else {return uri;}
        } else {
            if (uri.match(re)) {return uri.replace(re, '$1' + key + "=" + value + '$2');
            } else {
                var hash =  '';
                if( uri.indexOf('#') !== -1 ){hash = uri.replace(/.*#/, '#');uri = uri.replace(/#.*/, '');}
                var separator = uri.indexOf('?') !== -1 ? "&" : "?";
                return uri + separator + key + "=" + value + hash;
            }
        }
    };
    var themeUrl = window.location.href;
    for (var key in themeQueryParam) {
        themeUrl = updateQueryStringParameter(themeUrl, key, themeQueryParam[key]);
    }
  
        window.location.href = themeUrl;
        

    }
})();

• In the editor, go to line number 6 and change the preview_theme_id to your new theme id.

• Click on the Save Changes button.

• Click on the 'Finalize & Exit' button.

Targeting audience

You can target a Shopify theme to specific audience segments. For example, you can restrict the Shopify theme to visitors coming from a particular campaign. Alternatively, you can target it at all visitors.

• Click on the 'Add Rules' button.

• Choose the audience category you wish to target for the split URL test from the popup.

• Then, from the attribute dropdown, select one of the attributes. For the above example, the audience category will be geography, and the attribute will be country.

• Then, from the operation dropdown, select one of the operations, such as 'is one of' or 'is not one of' based on your requirement. For the above example, the operation type will be 'is one of'.

• Then, provide the value in the input field. For the above example, the value will be the United States. You can enter multiple values. In a few of the cases, you can see the drop-down option under the values section.

• Finally, under the targeting behavior dropdown, select the percentage rollout option. Then configure the value as 50% for the default and 50% for the newly created behavior.

• Once the target audience is configured, make sure to click on the 'Save' button.

Add goals

In the experience goals tab, perform the below steps to add a goal.

• Click on the 'Add goals' button.

• In the popup, select the type of goal event you want to add.

• Then select the goal event that you want to add.

• Enter the access code and click on the 'Add' button.

• If you don't find your goal event, then you can create a new goal event by clicking on the 'Create a new goal' option.

By default, the first goal event that you add will be considered the primary goal. If you have multiple goal events, then you can make any one of the goal events the primary goal event.

Preview of the split URL test

Once a split URL test is configured, make sure to preview it before taking it live.

• Click on the 'Preview' icon present on the behavior.

• Then from the popup select the Version A page URL.

Instead of opening Version A page you will be redirected to Version B page. If you see this behavior then your split URL test has been configured properly.

If you do not see the changes in the preview, then check if the CustomFit code snippet is installed on the website. If you have installed the code snippet, then check whether your browser is blocking the opening of new tabs.

Taking a split URL test live

To take the split URL test live click on the power switch toggle.