# How to analyse thousands of open-ended responses with AI: a method with validation checks

> A practical workflow for AI-assisted coding of open-ended responses, including codebooks, held-out evaluation and reproducible counts.

Source: https://www.pulselake.co/blog/how-to-analyse-thousands-of-open-ended-responses-with-ai-a-method-with-validation-checks
By PulseLake Team · PulseLake

To analyse open-ended responses with AI, define the research question, build a reviewable coding scheme, evaluate it on held-out responses and calculate results from the coded dataset. Keep every theme connected to its source text. The method below describes how to validate the work; it does not claim that a particular model or accuracy level has already been validated.

The temptation with a large file is to ask for a summary immediately. That can produce a useful first impression, but it leaves basic questions unanswered: what was counted, what was missed and whether the categories mean what the team thinks they mean.

![Open-ended comments flow through AI coding into themes, followed by validation checks.](https://www.pulselake.co/blog/img/production/0df60e9f6648f3a07d568f434d4cbbc4d31cac07-1200x840.png?w=1600&fit=max&auto=format)

## 1. Define the decision and the unit of analysis

Start with the decision the comments will inform. A team investigating checkout problems needs different categories from a team measuring brand associations, even if both datasets contain the word “easy.”

Then define the unit. Will the analysis count respondents, individual answers or passages within answers? Can one answer receive several codes? These choices determine how percentages and comparisons should be interpreted.

Keep a stable response identifier and the original text. If a response contains several passages, retain the link to its parent record. Without that link, later aggregation can accidentally count one person several times.

## 2. Prepare the data without erasing meaning

Remove records only under documented rules. Empty responses, duplicates and test submissions may require different treatment. Preserve a record of exclusions so another analyst can reproduce the final dataset.

Retain language, question wording and relevant collection context. A short response may be meaningful only alongside the question it answered. Translation can help analysis, but keep the original and record the translation process.

Apply the project's approved handling of personal information before sending text to an external service. The appropriate workflow depends on the data and permissions; a convenient upload interface does not establish that the upload is permitted.

## 3. Read a varied subset before defining themes

Review material from different segments, time periods and response lengths. The aim is to understand the range of answers and identify issues that a purely frequent-word approach could miss.

Use AI to suggest candidate themes if helpful, but have a researcher inspect the underlying passages. A broad label such as “service” can hide different causes: waiting time, staff knowledge or an unresolved complaint.

Keep an uncertainty category. Forcing every response into a confident label makes the output tidy at the cost of honesty. Ambiguous records are useful inputs to the next review round.

## 4. Write a codebook someone else can use

For each code, include a definition, inclusion rule, exclusion rule and example. If codes overlap, explain when multiple coding is allowed.

Here is a fictional illustration for a delivery study:

| Code | Include | Exclude |
| --- | --- | --- |
| Late arrival | The item arrived after the expected time | A complaint only about tracking information |
| Tracking uncertainty | The person could not determine shipment status | A late arrival with no tracking concern |
| Damaged item | The item was described as physically damaged | Dislike of the product's appearance |
| Unclear | The meaning cannot be established from the response | A clear negative comment that lacks strong emotion |

A comment saying “It arrived late and the tracking never updated” can receive both the first and second codes. That is a feature of the scheme, not a duplicate to remove.

## 5. Separate development from evaluation

Use one set of responses to refine the codebook and another held-out set to evaluate the resulting process. If the team repeatedly changes the instructions after inspecting the same evaluation examples, those examples become part of development.

Have qualified reviewers establish reference coding and resolve disagreements. Human agreement is not automatic ground truth; disagreements can reveal ambiguous definitions or genuinely mixed responses.

Evaluate categories individually. An overall score can hide poor handling of an uncommon but consequential issue. Decide in advance which errors matter most for the business decision and which categories require manual review.

## 6. Inspect errors before scaling the run

Look at false positives, missed cases and records placed in the uncertainty category. Ask whether the problem comes from the codebook, missing context, language handling or the model's interpretation.

Do not improve apparent performance by quietly removing difficult records. If a class of responses cannot be handled reliably, route it to manual review or narrow the approved use of the output.

Record the model, instructions, codebook and processing settings. Reproducibility is especially important when the work runs in batches and a later batch uses a different configuration.

## 7. Calculate from records, not generated prose

Once coding is accepted, calculate counts and proportions with an explicit data process. Python with [pandas grouping operations](https://pandas.pydata.org/docs/user_guide/groupby.html) or [R](https://www.r-project.org/) can support reproducible aggregation. The choice of language matters less than clear definitions and retained code.

In a hypothetical study, 40 of 200 respondents mention late delivery. That is 20% of those respondents. If some also mention tracking, the combined number of topic mentions can exceed the number of people; the report should not call those mentions additional respondents.

Keep denominator rules beside the results. State whether the base is everyone who completed the study, everyone shown the question or everyone who supplied a usable answer.

## 8. Interpret patterns in context

Frequency is not the same as importance. A rare issue can be consequential, and a common phrase can be too vague to guide action. Read representative and contradictory passages before recommending a response.

Compare segments only when the data supports the comparison. Differences in who answered, question exposure or collection timing can create apparent patterns that need further investigation.

Use quotations carefully. Select text that accurately illustrates the theme and preserve relevant context. Do not manufacture a cleaner quote or combine several participants into one fictional statement.

## Which tools can support the workflow?

[NVivo](https://lumivero.com/products/nvivo/), [ATLAS.ti](https://atlasti.com/) and [MAXQDA](https://www.maxqda.com/) are qualitative-analysis tools to consider. [Thematic](https://getthematic.com/) is relevant to feedback analysis, while broader research platforms such as Qualtrics may connect analysis to collection.

PulseLake is a research operating system that connects study design and evidence collection with structured analysis, interpretation and reporting. Its research frameworks organise questions and measures into dimensions and sub-dimensions. That structure gives the evidence a shared meaning and makes it usable for analysis beyond an individual response or survey.

For qualitative work, the important connection is between the source evidence and the concepts being analysed. A dimension-based framework can make the intended interpretation explicit; coding quality and reliability still need their own review.

Ask any supplier to demonstrate source retrieval, code revision and export on your material. A list of AI features does not establish that the workflow supports the validation process you need.

## What should the final report disclose?

Include the research question, data source, exclusions, coding scheme, model-assisted steps and review process. Describe important error patterns and any categories handled manually.

Keep the coded dataset, codebook version and calculation process with the report. A later reviewer should be able to reproduce the numbers and inspect the evidence behind a claim.

If the dataset changes or the model is upgraded, assess whether the earlier validation still applies. The result of one evaluation is not a permanent licence to use the method for every future study.

## Frequently asked questions

### How many responses should humans review?

Choose the review design based on the diversity of the data, category frequency and consequences of errors. There is no universal count that validates every dataset or method.

### Can one response have multiple themes?

Yes, if the codebook allows it. Keep respondent counts separate from theme mentions and explain how percentages are calculated.

### Is a high overall accuracy score enough?

No. Inspect performance by category and relevant segment, especially for uncommon issues that matter to the decision. Review the kinds of errors, not only the aggregate score.

### Can the AI write the final recommendation?

It can help draft wording, but a researcher should check the evidence and reasoning. The recommendation must stay within what the study and its validation can support.

## Related reading

- [Research resources](https://www.pulselake.co/resources)
- [Contact PulseLake](https://www.pulselake.co/contact)