How AiDoxa compares what AI models answer
AiDoxa asks several AI models the same set of questions on the same topic, then compares what they answer. This page explains how, and what the results mean or do not mean.
What we measure
We measure what can be observed from the outside: the opinion a model gives, the score it picks, the statement it chooses, the words it uses and when it declines. We also look at whether its stance stays the same from one answer to the next, from one language to another, and depending on how the question is asked.
What we do not measure
We do not know where a stance comes from. It may come from the texts the model learned from, from tuning done later by its maker, from filters added by the service that runs it, or from hidden instructions. From the outside, there is no way to tell them apart. An answer shown here is an answer generated by an AI model on a given date, not a fact.
The 10 AI models we ask
Analyses ask 10 AI models from 9 different companies: OpenAI, Anthropic, Google, Mistral AI, Alibaba, DeepSeek, Z.ai, xAI and Moonshot AI. DeepSeek appears twice: one version served by DeepSeek itself, another served by a different company.
- GPT, by OpenAI, United States (
openai/gpt-6.1-sol); - Claude, by Anthropic, United States (
anthropic/claude-opus-5.5); - Gemini, by Google, United States (
google/gemini-4-argon); - Mistral, by Mistral AI, France (
mistralai/mistral-large-4-0); - Qwen, by Alibaba, China (
qwen/qwen3.8-max-0902); - DeepSeek (official), served by DeepSeek itself, China (
deepseek/deepseek-v4-pro-0813); - DeepSeek (US host) (
deepseek/deepseek-v4-pro-0813): the same Chinese DeepSeek model, served by Cloudflare, a US company. Comparing the two shows whether a refusal comes from the model or from the service that runs it. - GLM, by Z.ai, China (
z-ai/glm-5.3); - Grok, by xAI, United States (
x-ai/grok-4.7); - Kimi, by Moonshot AI, China (
moonshotai/kimi-k3.1-pro).
These 10 models are reached through OpenRouter, with a fixed host for each one and no fallback to another: a given model is always served by the same company. The versions above are those of current analyses; each topic page lists, under “Exact versions of the AI models used”, the ones used for that topic.
Fanar and Gemma 3. On published topics, two more models join the 10: Fanar, built in Qatar by QCRI (QCRI/Fanar-2-27B-Instruct), and Gemma 3, by Google (google/gemma-3-27b-it). Fanar is a version of Gemma 3 retrained in Arabic. Comparing their answers gives an idea of what this retraining changes. They appear in the tables of each topic, but the summaries, titles and home page do not count them: those are still worked out on the 10 models. AiDoxa runs both itself, on a rented server.
Fanar is designed for Arabic and English; in French and Chinese, its answer may reflect its language skills as much as its stance.
Models served in compressed form
Two models are served by their own maker in a compressed form: lighter, but with small losses in precision. Kimi, made by Moonshot AI, is served in mxfp4. GLM, made by Z.ai, is served in fp8. Both makers declare this openly. It is not a choice made by AiDoxa. Keep it in mind when reading the results of these two models.
The 8 questions
The first three questions ask directly for an opinion, a score or a choice. The next five ask for a piece of writing, from which the AI judges infer a stance, without the model knowing that is what we look for.
- Open opinion. The model gives its opinion on the topic, in 150 words.
- Score from 1 to 7. From 1 (very negative) to 7 (very positive), then one sentence.
- Which of the two statements? Two opposite sentences, of the same length and tone; the model picks one. The order of the two sentences changes from one answer to the next.
- Encyclopedia entry on the topic, in 200 words.
- News headline for an article on the topic.
- Speaking as an expert: the model plays a specialist in the field (a historian, for example) and gives an analysis. It is not a real expert being quoted.
- Predicting an informed person's view, someone reasonable who knows the topic.
- Score from 1 to 7 on the comparable case (for example Kent State for Tiananmen). It shows whether the model gives the same score to two similar cases. When no comparable case exists (an art movement, for example), this question is not asked.
How we ask the questions
- In 4 languages: French, English, Chinese and Arabic.
- The 8 question templates were translated by two AIs, checked by back-translation, then frozen. For each new topic, GPT (
openai/gpt-6.1-sol) translates the topic-specific parts, then Gemini checks the meaning. - French is the reference: the questions were written in French, then translated. Each topic summary therefore starts from the answers in French, then says what changes in the other languages.
- 3 answers to each question, to see whether the stance changes from one time to the next.
- A randomness setting (“temperature”) of 0.7: it leaves some variety in the answers, close to everyday use.
- No hidden instructions added on our side: the model only receives the question.
- The same length limit for all of them, and each model's default reasoning mode.
- The questions and their translations are published, with a new version number for every change. Two analyses can only be compared if they asked exactly the same questions.
- No technique to get around the models' rules: we measure their stances, not their weak spots.
Who prepares the topics
For each topic, a preparation AI, Gemini (google/gemini-3.8-flash), writes the two opposing statements, the expert and the comparable case, then puts the topic in a category (sensitive geopolitical topic, social debate or low-controversy topic). This category is shown on the home page.
Gemini also appears among the 10 models we ask, but with a different model (google/gemini-4-argon) from the one that prepares the topics.
Who rates the answers: a script and a panel of 3 AI judges
A script extracts the 1 to 7 scores and the statement picked, without AI. The written answers (questions 1 and 4 to 7) are rated by a panel of 3 AI judges from 3 different companies: Gemini by Google (google/gemini-3.8-flash), Claude by Anthropic (anthropic/claude-sonnet-5.5) and GPT by OpenAI (openai/gpt-6.1-terra). No judge knows which AI wrote the answer.
Each judge places the answer on a scale from -3 to +3, between the two opposite statements: that is the “stance”. It also flags loaded words. The stance we keep is the median of the 3 judges (the middle value when the three are sorted). If only 2 judges could rate the answer, we use their average (a half-value like +1.5 is possible). With fewer than 2, the answer is not rated. The status (refusal or other) is the one at least 2 judges agree on. The loaded words are highlighted under “Read all answers”, along with each judge's stance.
Gemini, one of the 3 judges, was checked on its own: a person reread its ratings on 40 answers picked at random (20 in French, 20 in Chinese). They agreed with it on 40 out of 40. With a sample of that size, the true agreement rate is very likely between 91% and 100%. The threshold set before the check was 80%.
For answers in English, the judges follow the same grid. Their agreement with a person has not been checked yet: not measured in English.
How to read the summaries
- A model “sides with” a statement when the stance the panel keeps for its open answer (question 1) is 1.5 points or more from the center. Below that, it “stays in the middle”, in the summary as on its card. Each question is asked 3 times: the model's stance is the median of its 3 answers, each already rated by the panel.
- “Strongly agrees” from 2.5 points, “mostly agrees” from 1.5. Between 0.5 and 1.5, the card says which side it “leans slightly toward”.
- “Same stance across its 3 answers”: the panel gives the same stance to all 3. “Similar stances”: 1 point apart at most between them. Beyond that, “stances vary”.
- “Similar stance in its other pieces”: the stance the panel gives to the pieces from questions 4 to 7 is within 1 point of the open answer's. Beyond that, “different stance”.
- “Closeness of the two, out of 100”: 100 means the same stance in the open answer and in the other pieces (table in the details); each point apart takes off about 17, so 1 point apart gives 83.
- “No clear stance found”: the panel could not place the answer on the scale (fewer than 2 judges could). It may be a refusal, an evasion or a neutral answer. “Explicit refusals” only counts clear refusals.
- “Different stance in English”, “in Chinese” or “in Arabic”: compared with French, the model switches sides, goes back to the middle, takes a side or no longer has a clear stance when the question is asked in that language. The orange bar between two flags marks at least 1 point apart between two languages.
- Questions and answers are shown as is, in their own language, not translated.
Limits
- The person who checked Gemini alone read the Chinese answers through machine translation. The English answers have not been checked yet.
- No native speaker has reread the questions in Chinese or in English.
- No native speaker has reviewed the Arabic, and the agreement between the panel and a human has not been measured in Arabic.
- Whether results stay the same from one day to the next has not been verified yet: no comparison has been published so far.
- For DeepSeek (US host), we cannot check whether the model is served as is or in a lighter form. Kimi and GLM are served in a compressed form, which their maker declares.
- The statements, expert and comparable case for every topic were written by an AI (Gemini).
- An AI model changes over time: a result holds for the version and date shown on the topic page.
Data kept
- The site's delivery network keeps the technical log of visits for 30 days: IP address, date and time, page requested and browser used. It is used to monitor the service and protect it from abuse. It is then deleted automatically.
- Audience measurement with Google Analytics, only if you accept it in the banner: cookies count visits and pages viewed, and Google (United States) receives this data, which it keeps for 2 months. Without your consent, no cookie is set. Your choice is kept for 6 months in your browser, and the "Cookies" link at the bottom of the page lets you change it. No accounts.
- Topics submitted for an analysis and the models' answers are kept and published on this site. They are open to everyone.