Experimentation is a first-class principle at Revenue HQ. Not an afterthought.
The desire to have an integrated testing suite that focused on maximizing profits was the first impetus behind building Revenue HQ, and it remains core to our corporate identity.
We support both frequentist (traditional A/B) and Bayesian (smart learning) testing mechanisms.
Every experiment type will have a single action that is taken to be the success metric (e.g., for landing pages, the success metric is creating a new Account). The success metric is independent of the testing methodology. Again, all landing pages use new Account creation as the success metric for the test, whether the landing page experiment is run in frequentist or Bayesian mode.
Bayesian tests use the Thompson Sampling methodology against a Beta distribution. Frequentist tests pick a random variant based on the individual weights of each variant.
Attributes:
| Name | Type | Description |
|---|---|---|
| methodology | string | One of: bayesian or frequentist. Default value: bayesian. |
| lookback_days | integer | How many previous days worth of pulls and conversions to consider when calculating conversion rates. This lookback window will also be enforced with Bayesian tests when sampling against the Beta distribution. |
| conversion_hours | integer | The maximum number of hours after the visitor is bucketed into a test to consider the conversion successful. Conversions that happen after this cutoff will be considered as failures when calculating conversion rates and when running Bayesian tests. |
| pulls | integer | Unique number of individuals that have been bucketed into the test (subject to the lookback_days window). |
| conversions | integer | Unique number of conversions of the visitors bucketed in the pulls attribute, where that conversion happened within conversion_hours hours after the pull occurred. The conversion rate is thus defined as: conversions / pulls. |
| reset_at | integer | Unix timestamp of when the experiment was last reset (in seconds since the Unix epoch). If set, only pulls and conversions that occurred after this timestamp will be considered. |
| created_at | integer | Unix timestamp of when the experiment was created (in seconds since the Unix epoch). |
| updated_at | integer | Unix timestamp of when the experiment was last updated (in seconds since the Unix epoch). |