This analysis examines AI-Assisted Acceptance Testing Still Cannot See and separates supported implications from claims that still depend on context or further evidence.
Read the evidence below as a decision trail: what changed, why it matters, which trade-offs shaped the result, and where the conclusion still depends on context.
AI helps you accept the program. The most dangerous thing is not when it makes an error, but when it says "passed". There are actually two levels of acceptance: functional that can be checked by machines (whether the file is present, HTTP 200, whether the text is correct), and aesthetic/behavioral that cannot be checked by machines (whether the video is actually played, whether the visuals are correct, and environmental differences). I encountered three blind spots in this revision, and came up with a rule: measure before drawing conclusions, and know which layers cannot be inspected by machines, and which layers must be inspected by humans.
AI acceptance is divided into two levels, and the machine can only inspect one level.
People who use AI-assisted development will almost always let it do the acceptance check: run a script to confirm whether the file has been sent, whether the page can be returned to 200, and whether the key text is correct. AI can do these things quickly and accurately, so that’s not the problem.
The problem is that there are actually two levels of acceptance. The first layer is functional, which can be verified by the machine: whether the file exists, whether the HTTP status code is correct, and whether the text content is correct. The second layer is aesthetic/behavioral, which machines cannot detect: whether the video is actually being played, whether the visual proportions are correct, and whether it looks different in different environments. The machine can only test the first layer, and the most dangerous thing is that when the first layer "passes", it will also give you a false confidence, making you think that the whole thing is OK.
This is not a complaint about "AI is unreliable". People who know how to use AI should not distrust it, but know clearly where it will give you false confidence, and design corresponding verification disciplines for that layer. The following three cases are all from the blind spots I actually encountered when the gwarket L5 version was launched.
Case 1: The screenshot deceived me because headless can’t decrypt the video
The background of the scene in the desktop version is an H.264 video. I used a headless browser (a browser with no interface and running in a pure background) to take a screenshot for acceptance. The screenshot looked "normal", so I thought I had passed this test.
But there is a trap here: the headless browser cannot decode H.264 video encoding at all. What it captures is actually the static fallback frame of the video, not the actual video playing. In other words, "the screenshot looks normal" does not mean "the video will actually move." The only thing that passes the screenshot is "there is something in this position and the format is suitable." As for whether it will be broadcast smoothly, the screenshot cannot answer the question at all.
The conclusion of this matter is clear: for outputs such as videos and motion effects, the machine can only verify that "the file is sent and the format is correct." The "real broadcast" layer must be viewed by real people in a real browser. So when I delivered it, I directly stated "I can't test this item. I need you to confirm it with your own eyes." I didn't package it as if it had been tested. It is more reliable to honestly mark "I can't test it" than to pretend that everything has been tested.
Case 2: The intuition of "the characters become smaller" almost made me wronged CDN
My first reaction after going online was: Why is the text on the official website so obviously smaller? Intuition immediately points to Cloudflare or the deployment process, wondering whether the CDN (content delivery network) has done something in the middle, compressed or rewritten the style.
But instead of blaming the CDN first, I start by measuring three things:
- Online version and local version
/l5/style.cssComparing the file byte numbers, the results are completely consistent (44106 = 44106), which means that the CDN has not changed the CSS. - The computed font-size of the large standard measured using the browser: 1920px wide is 142px (exactly the maximum value of the design), 1280px wide is 95px, both are within the design range.
- Compare the viewport meta tag, online is consistent with the local one.
If all three are correct, it means this is not a deployment bug. There are two real reasons: First, the big standard uses clamp(48px, 8.3vw, 142px) With this kind of responsive font level (the size of the font is tied to the width of the window), the font size becomes smaller as the window becomes narrower; second, the browser paging itself is scaled. To sum up this blind spot in one sentence: When you feel that "all words become smaller together in equal proportions", it is almost caused by browser scaling (just press Ctrl+0 to reset), not a program error.
The real lesson here is not how to write CSS, but the order of attribution. The most opaque layer (CDN, build pipeline) is the easiest to use as a scapegoat, because you can't see it, so it's easy to doubt. But the correct approach is to weigh the evidence first and then draw a conclusion, rather than passing the blame to the link you are least familiar with first.
Case 3: The two videos are the same size, but they are not the same one
Before uploading the scene video, I found that the file size of a homemade video and a clip in the reference material library were exactly the same, both 3.64MB. The first instinct is: Could this be the same file? If so, it is a copyright risk and must be blocked.
But instead of making a judgment just by looking at the file size, I used multiple independent signals to cross-check:
- Use ffprobe (video information detection tool) to compare the size and length: the homemade one is 16:9, 8 seconds; the reference one is 1:1, 4 seconds. Completely different.
- Extract the content of the frame ratio pictures: one is a spaceship cabin looking at the sea of clouds, and the other is a 3D rendering of industrial style. Completely two different things.
The conclusion is that the two videos are completely different, and the identical 3.64MB is purely a coincidence after compression. The judgment point in this case is: the files are of the same size and are not equal to the same file. Provenance (source determination) cannot be determined by relying on a single weak signal. Multiple orthogonal (independent of each other and will not be wrong together) signals must be used for cross-verification.
Two misjudgments, the same antidote
Looking at these three cases together, you will find that they are actually two different modes of miscarriage of justice.
The first is "the verification method itself has a blind spot": Case 1 was deceived by screenshots, because the verification tool I used (headless screenshots) was inherently unable to verify the layer I wanted to verify (whether the video was played or not). The second type is "attributing the blame first if there is no evidence": in the second case, I want to blame CDN, and in the third case, I want to treat it as the same company. In both cases, they have not yet obtained the evidence, so they intuitively assign the problem to a suspicious-looking object.
The common antidote to these two misjudgments is the same sentence: measure first, then draw conclusions. Measuring first, forcing you to produce verifiable evidence, cures the second type of misjudgment; and "knowing which level your verification method cannot test" cures the first type - when you know that the screenshot cannot be verified for playback, you will take the initiative to make up for manual acceptance, instead of being fed false confidence by a seemingly normal screenshot.
What this means
In my opinion, the most worthwhile investment in AI collaboration is not "let AI help you do more inspections", but "clearly distinguishing which layers it can inspect and which layers it cannot." The former is just outsourcing manpower, while the latter is real discipline. The difference between a person who knows how to use AI and a person who is fed false confidence by AI lies in how clearly this line is drawn.
I also believe that the value of the discipline of "measure before drawing conclusions" does not lie in how profound it is, but in that it is anti-human. Human intuition naturally likes to find a suspicious object to blame—preferably the link that you are least familiar with and the most opaque, because it is the least laborious to blame. But the more times like this, the more you have to hold back, first measure the few hard numbers you can get (number of bytes, computed value, video size) and let the evidence speak for itself. In each of my three cases this time, if you follow your intuition, you will come to the wrong conclusion.
The last thing I want to say is that honestly stating "I can't test this level" is not a sign of weakness, but a sign of professionalism. What you are most afraid of is not that the AI fails to pass the inspection, but that you hand over "functional all passed" as "quality OK". As a result, the video is not broadcast at all and is discovered by users only after it is online. It is better to honestly write "This needs to be confirmed with your own eyes" when delivering it, than to use a beautiful screenshot to instill false confidence in yourself and others.
Practical questions and boundaries
Why is AI’s “functional acceptance passed” dangerous?
Because when the functional acceptance (file presence or absence, HTTP 200, text is correct) is passed, it will give you a false confidence that "the whole thing is OK", but it cannot check the aesthetic/behavioral layer (whether the video is actually played, visual proportions, environmental differences). The most dangerous thing is not that the AI makes an error, but that it says "passed" to stop you from asking questions, but it doesn't detect that at all.
What’s wrong with using headless screenshots to check videos?
The headless browser cannot decode video encodings such as H.264, and what is captured is a static fallback frame, not the actual video being played. So "the screenshot looks normal" does not mean "the video will move". For video and motion effect output, the machine can only verify the format and file. "Real broadcast" must be confirmed by a real person in a real browser.
The online screen is different from the local one, what should I do first?
Measure first, don’t blame the CDN or deployment first. Three specific steps: byte-compare online and local asset files (CSS/JS), measure the computed value in the browser (such as font-size), and compare viewport meta. If all three are consistent, it is not a deployment bug. The most opaque layer is most likely to be scapegoated, but it is usually innocent.
What to take away
The article's value is in the evidence and trade-offs behind what AI-Assisted Acceptance Testing Still Cannot See, not in treating the conclusion as universal.