How to set up an experiment (Experiment Engine)

Use the Experiment Engine to test variations of buttons, windows, invitations, and invite rules or rule sets. In more technical terms, an experiment is a targeted modification of any currently deployed HTML snippet/Website definition.

Note: This feature is only available for Bold360 Plus subscribers and Bold360 AI platform accounts.

Gain insight into your chat strategy by varying the chat button or invitation rule set that is presented/applied to customers who qualify for the experiment based on criteria of your choice. Experiments are active for each HTML snippet with which a website is associated.

For example, let's say you currently have an HTML snippet deployed to your home page that shows a yellow fixed-position chat button to your customers. You want to decide if a green fixed-position button will perform more effectively. To test this, you open the website associated with your HTML snippet and set up an experiment that shows the currently deployed yellow button to 50% of customers (also referred to as the Control Group) and the experimental green button to the other 50%.

Tip: Experiments are a convenient way to modify buttons and rule sets without re-deploying an HTML snippet.
  1. Open a website, as follows:
    1. In the Web Admin Center, go to Organization > Websites. The Websites page is displayed.
    2. Select an existing website or click Create New.

    Result: The New/Edit Website window is displayed.

  2. Click the Experiments tab.
  3. If you plan on running multiple experiments, please take a moment to understand the implications of running multiple experiments simultaneously. As a general rule, you should avoid setting up your experiments that allow a single visit to qualify for multiple experiments. Proper filtering should ensure that each customer will only qualify for one experiment. Nonetheless, if you are unsure of your filters, we recommend NOT selecting the Allow customers to qualify for simultaneous experiments option.
    • When Allow customers to qualify for simultaneous experiments is selected, each customer can potentially qualify for multiple experiments per visit (based on the Who to include filter). This may result in a single user being represented multiple times in reports. We recommend using this feature only when your filters allow customers to qualify for one active experiment at a time.
    • When this option is not selected, customers are randomly and evenly distributed between running experiments and checked against active filters. Customers not qualifying for an experiment will not be forwarded to another experiment (even if they would have otherwise qualified for the "next" experiment). If your experiments are unfiltered or use similar filters, do not allow customers to qualify for simultaneous experiments.
  4. On the Experiments tab, click New Experiment.

    Result: An experiment is created.

  5. Give the experiment a Name.

    The name cannot be changed once the experiment has started.

  6. Schedule the experiment, as follows:
    Option Description
    Scheduled Start Begin analyzing customers at this date and time. It can take up to a minute after the scheduled start time for buttons and invitations to be presented.
    End Stop applying experiment filters to customers at this date and time. Any customer who has already been presented a test item will still be evaluated.
    Add different close date (Scheduled Close) To stop evaluating the behavior of participants still in the experiment, you must close the experiment. Any customer who still participates in an experiment will not be included in the results after this time and date.
  7. If you do not want the experiment to apply to all customers, under Who to include you can define customers who will be subjects of the experiment.
    1. Click Add Criteria and select from available filters.
  8. Under What to test, set up your experiment:
    1. Click Add Action and choose an item.
      Note: Control Group refers to the currently deployed buttons and rulesets for a given site (HTML snippet with a website definition).
    2. Choose a specific instance of the item.

      Example: ThisSpecificButton or ThatSpecificRuleset.

    3. Add additional items as required (Add Action button).
    4. Set the percentage of customers included in the experiment who will be engaged by each item in the experiment.

      Example: Show ThisSpecificButton to 33% of qualified customers and ThatSpecificRuleset to 33% of qualified customers, and show the currently deployed item (Control Group) to 33%.

    Test Items Notes
    Fixed Button For HTML snippets already using a fixed chat button.
    Floating Button A floating button experiment hides a fixed button only when there is no other active floating button in the current deployment.
    Invite Rulesets For invitations, experiment filters are secondary to filters in the currently deployed invitation. (For example, if your current invitation is filtering for the USA, customers who are not in the USA will never see experimental invitations regardless of filters set in the experiment).
    No Interactions Choose this to show "nothing" in place of the currently deployed item.
  9. Save your changes.

    Result: The settings are applied to the website definition.

At the scheduled start time, for each HTML snippet to which this website is associated, customers will be analyzed and buttons/invitations presented. Customers can only be presented one item from the experiment at a time.
Remember: A running experiment cannot be modified.
Related Articles