You stop drinking coffee after noon, and that week you sleep better. Did the change help? Maybe. Or maybe work was quieter, the evenings were cooler, or it was simply a good week. One week can’t tell you.
Inside that everyday question is a real scientific problem: how do you separate the effect of one change from everything else going on in your life? In our first post we wrote about why we are building N1 Trial. This one is about the method behind it, the N-of-1 trial, and how the app puts it to work. At the end there is a more technical section for anyone who wants the details.
You are the comparison
In an N-of-1 trial, the “one” is you. Instead of comparing a group that makes a change with a group that doesn’t, you compare yourself with yourself. You live with the change for a few days, then without it for a few days. We call each of those stretches a block. You switch between blocks with and without the change several times, while measuring the same thing all the way through. [1]
A single good week proves little. If you sleep better every time the coffee stops at noon, and worse every time it doesn’t, that pattern is much harder to explain away.
The answer is about you, under the conditions you tested, in the weeks you tested them. It doesn’t automatically apply to other people, and it may not hold for the rest of your life.
Start with a question you can answer
“Can I sleep better?” is a good reason to start, but a trial can’t answer it. “Does stopping coffee at noon change how rested I feel in the morning?” is a question it can answer. It names one change and one thing to measure.
That one thing is your main measure, and you choose it before you begin. This matters more than it sounds. Your watch might track how long you sleep, how deeply you sleep and your heart rate at night. Look at enough numbers at once and at least one of them will almost always look better, just by chance. If you only decide afterwards which number counts, it is easy to convince yourself of an effect that isn’t there.
In N1 Trial you plan this in a short conversation with an AI assistant, or you start from one of the ready-made trials. Either way you end up with a plan: what you will change, what you will measure, and on which days.
A fair comparison
A few details in the plan do most of the work.
The order is random. If you choose when to make the change, you might pick the calm weeks without meaning to. N1 Trial draws the order of the blocks at random, with a balanced number of each and never more than two of the same kind in a row. Very short plans simply alternate, and a change with long after-effects starts with your usual routine.
A change needs time to settle. It can take a while to kick in and a while to wear off. So the first day or days of a block that switches condition are set aside: you still record them, but they don’t count in the comparison. How many depends on the kind of change, from none for something that works quickly to a week for a supplement that builds up in the body. This lowers the risk of one block spilling into the next, though it can’t rule it out.
You measure the same way every time. The same question at the same time of day, or the same device. N1 Trial asks you the question each day, or reads the measurement from your smartwatch, Oura or Whoop in the background.
Days you miss stay missing. The analysis doesn’t fill them in. If you tend to skip the worst days, though, the data will look better than your weeks really were.
Reading the answer
While the trial runs, you can follow your daily readings, but the app waits until the end to tell you what they add up to. If it told you halfway that the change was probably working, that could easily change what you record the next day.
When the trial is over, N1 Trial estimates the difference between the blocks with and without the change. The analysis allows for your days drifting over time and for one day tending to resemble the next, so a run of similar nights counts for less than the same number of unrelated ones.
Before you start, you can also decide how big an improvement would be worth it to you. In the coffee example, that might be one point more on a 0 to 10 scale of how rested you feel. An improvement you would barely notice may not be worth changing your habits for.
The result opens with a small card. It tells you the chance that your measure moved the way you wanted, the most likely size of the change, and, if you set a target, the chance that the change reached it. Under the estimate, a bar shows where the real effect most likely lies. The dot is the best estimate, the solid line is no change, and the dashed line is your target. Better is always to the right, and the narrower the bar, the more certain the answer.
Here are three made-up results from the coffee example, each with a target of one point.
Graphs and the full numbers are one tap away: your days with and without the change, and how the analysis arrived at its estimate. If there wasn’t enough to go on, for example because there were too few days, too few switches between the conditions or too many missed check-ins, the app says so in plain words instead of drawing a conclusion.
What a trial on yourself can’t do
Some limits come with the method, however carefully it is carried out. [2]
- You know which block you are in. If you expect the change to help, that can colour how you feel and what you record, especially when you rate something yourself.
- Something else in your life can happen to line up with one of the conditions. A random order makes that less likely, but can’t rule it out.
- Some changes last long after you stop, like learning a skill or losing weight. Switching back and forth doesn’t work for those.
- It isn’t a way to test medication or the treatment of an illness. Decisions like that belong with your doctor.
Where we are
The method is well tested. Doctors and researchers have used N-of-1 trials for decades, and the Bayesian models behind our analysis are described and used in the research literature. [3] We have also checked our own calculations carefully: on made-up data, the app’s analysis gives the same answers as independently written statistical software.
What’s new is making the method something you can use yourself, in an ordinary week. That is what we are learning most about right now, together with our first testers.
A closer look at the analysis
This part is for readers who want the statistical detail. You don’t need it to use N1 Trial.
Models. The model follows the main measure. Measurements that vary around an average, including 0 to 10 ratings, use a Gaussian model. Positive, skewed measurements use a Gaussian model on the log scale. Yes/no outcomes use logistic regression.
The Gaussian models include a linear time trend and AR(1) errors, which let neighbouring days be correlated and use the actual gaps between observations, so missing days need no imputation. Alternating-day designs also get a weekend term. The yes/no model has neither autocorrelation nor a weekend term yet. Carryover has no parameter of its own; the design handles it, through the set-aside days and the choice of starting condition.
Estimates. The analysis is Bayesian, with explicit priors scaled to the measure’s range, and it is computed on a deterministic grid rather than by sampling. The estimate is the posterior median, with an equal-tailed 95% credible interval: given the data, the model and the priors, the effect lies in that range with 95% probability. The interval says nothing about the spread of your daily readings, and it can’t correct for biases such as expectation. [3]
When the data aren’t enough. If one condition has no usable days, or either has fewer than two, there is no estimate at all. With only two usable condition periods, or fewer than 60% of the expected outcome check-ins answered, the app still shows the numbers but says the trial can’t answer the question.
Sensitivity. Every final result is estimated again with an effect prior 100 times wider, and trials with three blocks also with a trend prior four times wider. These show how much the result leans on the assumptions, and the app says in plain words whether a different assumption would change the picture.
Checks and known gaps. On synthetic data, the engine matches an independent Python implementation, JAGS for the Gaussian models, and the nof1ins R package from Schmid and Yang’s group. One gap is still open: with very sparse yes/no data, the estimated probability can shift by a few percentage points depending on the resolution of the calculation grid.
Sources
- AHRQ (2014). Design and Implementation of N-of-1 Trials: A User’s Guide. An introduction to the method, the questions it suits and the design choices involved.
- Shamseer et al. (2015). CONSORT extension for reporting N-of-1 trials (CENT) 2015: Explanation and elaboration. How to describe the design, measurements, analysis and limits of an N-of-1 trial.
- Schmid and Yang (2022). Bayesian Models for N-of-1 Trials. Bayesian inference for single trials and series of trials, including methods beyond those N1 Trial uses.
These publications describe the method. They don’t endorse N1 Trial.