You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b58c408
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: specs/abtesting-v3/common/schemas/Variant.yml
+75-5Lines changed: 75 additions & 5 deletions
Original file line number
Diff line number
Diff line change
@@ -53,12 +53,14 @@ metricResult:
53
53
properties:
54
54
name:
55
55
type: string
56
+
description: Metric name. Revenue per search results use `revenue_per_search`.
56
57
updatedAt:
57
58
type: string
58
59
description: Date and time when the metric was last updated, in RFC 3339 format.
59
60
value:
60
61
type: number
61
62
format: double
63
+
description: Metric value. For `revenue_per_search`, this is the winsorized mean revenue per search in the specified currency.
62
64
valueCIHigh:
63
65
type: number
64
66
format: double
@@ -78,10 +80,12 @@ metricResult:
78
80
pValue:
79
81
type: number
80
82
format: double
81
-
description: PValue for the first variant (control) will always be 0. For the other variants, pValue is calculated for the current variant based on the control.
83
+
description: |
84
+
P-value for this variant compared to the control.
85
+
Omitted when no p-value is available for this metric.
82
86
dimension:
83
87
type: string
84
-
description: Dimension defined during test creation.
88
+
description: Dimension defined during test creation. For revenue metrics, including `revenue_per_search`, this is the currency.
85
89
metadata:
86
90
$ref: '#/metricMetadata'
87
91
criticalValue:
@@ -95,20 +99,21 @@ metricResult:
95
99
type: boolean
96
100
description: |
97
101
Whether the pValue is significant or not based on the critical value and the error correction algorithm used.
Bayesian inference results for this variant metric.
144
+
Omitted when Bayesian results aren't requested or no Bayesian result is available for this metric.
145
+
Individual inference fields can be omitted when their values aren't available.
146
+
properties:
147
+
probabilityToBeBetter:
148
+
type: number
149
+
format: double
150
+
description: Probability that this variant is better than the control.
151
+
example: 0.991
152
+
relativeEffectCILow:
153
+
type: number
154
+
format: double
155
+
description: Lower bound of the 95% credible interval for the relative effect (variant/control minus 1).
156
+
example: 0.04
157
+
relativeEffectCIHigh:
158
+
type: number
159
+
format: double
160
+
description: Upper bound of the 95% credible interval for the relative effect (variant/control minus 1).
161
+
example: 0.39
162
+
evidence:
163
+
$ref: '#/metricEvidence'
164
+
165
+
metricEvidence:
166
+
type: object
167
+
description: |
168
+
Evidence status for this variant metric's comparison with the control.
169
+
Only present when the API has an evidence status for the comparison. Omitted when evidence isn't applicable or no status exists.
170
+
properties:
171
+
status:
172
+
$ref: '#/evidenceStatus'
173
+
required:
174
+
- status
175
+
176
+
evidenceStatus:
177
+
type: string
178
+
description: |
179
+
Whether this variant metric's result is trustworthy yet.
180
+
This doesn't mean the A/B test has ended, and it doesn't declare a winning variant.
181
+
182
+
- `enough`. Enough evidence has accumulated to trust this metric's result for this variant.
183
+
- `not_enough`. Evidence was computed but hasn't reached the threshold yet.
184
+
- `collecting`. Evidence is still accumulating, so it's too early to report a verdict.
185
+
- `no_data`. No usable data for this variant metric, and none is expected for it.
186
+
- `unavailable`. Evidence can't be computed for this variant metric's comparison.
187
+
example: not_enough
188
+
enum:
189
+
- enough
190
+
- not_enough
191
+
- collecting
192
+
- no_data
193
+
- unavailable
124
194
125
195
variantMetadata:
126
196
type: object
@@ -137,7 +207,7 @@ metricMetadata:
137
207
type: number
138
208
format: double
139
209
description: |
140
-
Only present in case the metric is 'revenue'.
210
+
Only present for `revenue` metrics.
141
211
It is the amount exceeding the 95th percentile of global revenue transactions involved in the AB Test. This amount is not considered when calculating statistical significance.
142
212
It is tied to a per revenue-currency pair contrary to other
143
213
global filter effects (such as outliers and empty search count).
0 commit comments