Skip to content

Design My Own Experiment

Read by connected AIs with get_playbook and id design-experiment.

Your own test, your metric and design, judged with every post’s numbers.

Use when they want to design, run or check their own experiment: A vs B, a metric they choose, their own rules. Not for the guided weekly test from a check-in (growth-check-in).

The user might say: “I want to design my own experiment: test [your idea] and judge it on [views, replies, bookmarks, profile clicks or follows].”

Follow this guide step by step.

The user wants to design and run their own experiment. They lead; you bring rigor and Xholic’s data. Don’t push a test they didn’t ask for, and don’t create anything until they agree to the design.

  1. See what’s already running or proposed with run_read_action “mentor.experiments”, and what they told Xholic with find_memory. If what they want to learn isn’t clear yet, ask in one question.
  2. Ground it in their numbers: run_read_action “mentor.account_data” with {“days”: 60} (posts with text and numbers). Say how many comparable posts they have and how much their numbers usually swing, because that decides how many posts the test needs.
  3. Recommend a design and say why, then let them change it:
    • metric matching the goal: impressions for reach, replies for conversation, bookmarks for saves, profile_clicks for profile interest, follows for growth (an estimate: X doesn’t say which post brought a follower);
    • one group against their own median, or A vs B (groupA, groupB, compareTo group_b) when they’re choosing between two options; a format or a date range when that’s the fair comparison;
    • posts per group (3 when the effect should be big, 5 or more when their numbers swing a lot), settle hours (48 is a good default), win and lose bars (1.2× and 0.8× by default), an end date if they want one.
  4. Only when they agree, call run_write_action “mentor.propose_experiment” with title, hypothesis and design; add “start”: true only if they said to start now.
  5. Explain how posts join: schedule each post, then run_write_action “mentor.update_experiment” with its scheduledPostId (or tweetId once posted) and group “a” or “b”. Offer to draft the posts for each group.
  6. For progress or a verdict, run_read_action “mentor.experiment_results”: walk through every post’s number, each group’s median and spread, and the reasons. Say plainly when it’s too early or when one post carries the result, and what would make it firmer.
  1. What’s already running or proposed. Call get_experiments.
  2. Their posts with text and numbers: how many are comparable and how much the numbers swing decides how many posts the test needs. Call get_account_data with {"days":60}.
  3. Recommend a design and why, and let the user change it: the metric for their goal (impressions, replies, bookmarks, profile_clicks, or follows as an estimate), one group against their median or A vs B, posts per group, settle hours, win and lose bars, an end date. Create nothing until they agree.
  4. Only when they agree, set it up with the design. Call propose_experiment with {"title":"<test>","hypothesis":"<what should happen and why>","design":{"metric":"impressions","groupA":{"label":"<A>"},"groupB":{"label":"<B>"},"compareTo":{"kind":"group_b"},"minPosts":3}}.
  5. Link each post to its group after scheduling it. Call update_experiment with {"experimentId":"<id>","scheduledPostId":"<post","group":"a"}.
  6. Progress or the verdict: every post’s number, each group’s median and spread, and the reasons. Say when it’s too early or one post carries it. Call experiment_results with {"experimentId":"<id>"}.