Nicasia Digital Solutions · Field notes

What tracking 146 homes on workers' phones taught us

2026-09-30T00:00:00+08:00

One housing project, 146 homes: 114 in the middle of their rows and 32 on the corners. Each home gets a set of metal items, a main gate, a side gate on the corners, railings, louvres on some house types, and every item passes through seven steps: measuring on site, ordering steel, fabrication, grinding and sealing, primer, installation and touch-up.

The workers report each step from their phones, in Chinese, Bengali or Burmese. Their app runs inside Telegram, the records sit in a cloud database, and the owner follows every block and every home on a web dashboard without making a call. We built it for our own metal fabrication floor and it has been in daily use since.

Most of what we learned in September 2026 came from four mistakes, all of them our own.

A block of ten homes should offer ten

On 4 September the owner opened the screen that assigns work in the factory. The quantity list offered 1, 2, 3, 4, 5, 6, 8, 10, 12, 20, 30 and 50. The block he had picked, B12, has ten homes. The item was louvres, which go on every home in that block, so the true maximum was ten. For the side gate, which only corner homes take, it was two.

He asked why the list was not built from the database. He was right. Every number the list needed was already in it, and the screen that workers use to report finished jobs had been counting from it all along. The factory screen used a fixed list someone thought was close enough.

A wrong list is worse than an ugly one, because people act on it. An order for 30 louvres from that screen would have sent the floor to cut 30 and the steel to be bought for 30.

We changed three things that day:

The block list shows only blocks where the item fits, with the number of homes beside each name: "B12 · 10".
The quantity stops at what the records allow for that block and that item.
The server refuses the impossible order on its own. A list on a screen can be bypassed or out of date; the server is the last place that can say no.

We also split one message into two. "We cannot count this" and "this item is not counted per home" used to show the same thing on screen, a blank maximum. They are different facts, and the second one is normal: work for another company has no homes in our records. The screen now says which one it is.

The total matters as much as the count

The same day, a check of every list against the database turned up a bigger number problem. The database said louvres go on all 146 homes. The bill of quantities, the contract's own count, said 70. The difference, 76, is exactly the number of type A homes, and louvres belong to type B only.

That one field let workers report louvres on homes that never get them, and it put 76 jobs that did not exist into the project's total. When we corrected it, the project's progress moved from 4.2% to 4.7% overnight. Nobody worked faster. The total had shrunk to the truth.

If a progress figure is going to reach a client, check the denominator against the bill of quantities before anyone reads the percentage.

Two screens deciding the same thing will drift apart

On 17 September a worker on another project finished one of the two homes in his block, reported it, and could not report the second. Tapping the block again to add the second home cleared his selection instead. The server had the right data the whole time. The trap was in the screen.

While fixing it we checked every project for the same kind of trap, and found one the owner had not reported. The workers' app and the owner's dashboard each decide which items a home gets, in separate code. After the louvres fix, the app judged from the house type recorded for each home. In the 146-home project the type was recorded for each block instead, so the app decided that none of the 70 type B homes took louvres, and nobody could report them. The dashboard looked fine. A hard-coded line there happened to give the right answer.

The fix was one rule, used by both: when a block has no type recorded per home, every home takes the block's type. Before shipping it we listed what the dashboard decided for every home in the five projects tracked per home, 2,209 pairs of home and item, and compared the list before and after. The only judgments that moved were those 70 louvres, and no screen on the dashboard changed at all. That comparison is now a tool we keep.

Old screens keep calling the new server

On 4 September the server's answer changed shape: the list of blocks went from plain names to name-and-count pairs, so the new screen could print "B12 · 10". We shipped the new screen at the same time and tested both together.

The owner's phone was not running the new screen. He had opened the app from an old Telegram message, and Telegram served him the page it had kept from before. That old page printed the new answer as text, and every block on his list read [object Object].

Nothing crashed. That was the danger. Our tests always pair the newest screen with the newest server; people in the field do not. Telegram only changes the version a message opens when a new message is sent.

The rule since then: a field the app already reads never changes shape. New information goes into a new field, and a new screen that cannot find it falls back to the old one and says what it does not know.

What to ask before you buy a site-tracking system

Where do its lists come from? Every choice a worker or supervisor taps should be counted from your project's records, per block and per home.
Does the server refuse what the records rule out, or only the screen?
Is the progress total built from your bill of quantities, and who checks it when the drawings and the bill disagree?
What happens when someone opens an old version of the app?
Can it tell "we do not know" from "this does not apply"?

We build these systems for other companies now, starting from their own records. A private review looks at how the work moves in yours.

Can ChatGPT and Claude read your website? How to check

2026-09-30T00:00:00+08:00

On 19 September 2026 we tested one of our own websites the way an AI crawler meets it. Its robots.txt welcomed every crawler. Googlebot, Bingbot, OAI-SearchBot and PerplexityBot got the page. GPTBot, ClaudeBot and CCBot got 25 bytes of plain text from Cloudflare instead: "Your request was blocked."

On 30 September we found a quieter gap on this site, nicasiadigital.com. Every crawler could open the pages, but our email address reached them as "[email protected]".

We had switched on neither setting. Both came with Cloudflare, and neither shows when you open the site in a browser: a browser gets the page, and it runs the script that puts the address back.

Which crawler does what

The companies behind the AI assistants each run more than one crawler, and each crawler feeds something different. Blocking one can cost you a great deal or nothing, depending on which one it is. This is what each company's own documentation says, as of September 2026.

Crawler	Company	What it reads your site for
OAI-SearchBot	OpenAI	Showing your pages in ChatGPT search answers
GPTBot	OpenAI	Training future models
ChatGPT-User	OpenAI	Opening a page because a user asked ChatGPT to
Claude-SearchBot	Anthropic	Improving Claude's search results
ClaudeBot	Anthropic	Training future models
Claude-User	Anthropic	Opening a page because a user asked Claude to
PerplexityBot	Perplexity	Listing and linking sites in Perplexity's answers; Perplexity says it does not train on them
Perplexity-User	Perplexity	Opening a page during an answer; it generally ignores robots.txt
Google-Extended	Google	A name in robots.txt with no crawler behind it: it decides whether Gemini may train on your pages or use them to ground answers
CCBot	Common Crawl	An open archive of the web that AI companies have used to train models

Two points in that documentation matter most. OpenAI's page says a site that blocks OAI-SearchBot stops appearing in ChatGPT search answers, while blocking GPTBot only keeps its pages out of training. Google's says Google-Extended has no effect on Google Search.

So on our site in September, ChatGPT search could still read us, because OAI-SearchBot got through. What we were losing was training: the next models would know less about the site.

Test it from your own computer

This loop asks for your home page under each crawler's name and prints the status code. It runs in the Mac Terminal and on Linux; on Windows, use Git Bash. Replace the address with yours.

for b in Googlebot bingbot GPTBot \
    OAI-SearchBot ChatGPT-User ClaudeBot \
    Claude-SearchBot Claude-User \
    PerplexityBot CCBot; do
  echo "$b $(curl -s -o /dev/null \
    -w '%{http_code}' -A \
    "Mozilla/5.0 (compatible; $b/1.0)" \
    https://www.example.com/)"
done

On 30 September at 11:05, Malaysia time, nicasiadigital.com answered 200 to all ten names. A 200 means the crawler gets the page. A 403 or a 503 means something in front of your site refuses it, whatever robots.txt says. If you would rather not open a terminal, our free crawler check runs the same test from a web page, along with the checks further down this note.

The test has one limit. It sends each crawler's name from your computer, and some firewalls also check where a request comes from, so the real crawler can be treated differently. Take a 200 as a good sign. Take a 403 seriously: on our site in September it matched a setting that turns away the real crawlers too.

Where Cloudflare keeps the switch

Since 1 July 2025, Cloudflare asks every new domain at sign-up whether to allow AI crawlers, and blocking is where it starts. Our domain moved to Cloudflare on 2 September 2026. Seventeen days later the training crawlers were still being turned away.

In the dashboard, open the domain, then Security, then Settings, and find the Bot traffic group. "Configure AI bot policies" has three rows: Search, Agent and Training. Ours read Allow, Allow, Disallow. Training was the row sending 403 to GPTBot, ClaudeBot and CCBot. We set it to Allow at 15:10 on 19 September, and the next run of the loop showed 200 for all nine names we tested that day.

At the bottom of the same panel, "Enable Bot Preference Sync" lets Cloudflare write its own lines into your robots.txt. We keep it off. The site has a robots.txt of its own, and two sets of rules would sooner or later disagree.

The email address crawlers could not read

Cloudflare's Email Address Obfuscation was on for this domain, and we had not turned it on. It replaces each email address in a page with a link to /cdn-cgi/l/email-protection and the words "[email protected]". A small script then puts the real address back in the visitor's browser, so people never see the swap.

A crawler that reads the HTML and stops there gets "[email protected]". A December 2024 study by Vercel and MERJ found that GPTBot, ClaudeBot and PerplexityBot fetch pages without running their JavaScript, so for them the address is simply gone.

We left the setting on and changed the pages. Cloudflare skips anything between  and , so our build now wraps every address in those two comments. Turning the feature off for the whole domain works too. Addresses inside <script> tags and in the <head> are never rewritten, which is why the address in our structured data was readable all along.

To check yours, fetch the contact page the way a crawler does and count Cloudflare's markers:

curl -s https://www.example.com/contact \
  -A "Mozilla/5.0 (compatible; GPTBot/1.0)" \
  | grep -c "email-protection"

Anything above 0 means at least one address on that page is hidden from crawlers. On our contact page it now prints 0.

What else a crawler misses

The same study found that Gemini, through Googlebot, and AppleBot do render JavaScript. The OpenAI, Anthropic and Perplexity crawlers it measured did not. Anything your site adds after its scripts run is missing for them: prices loaded by a script, a menu built in the browser, reviews from a widget, sometimes the whole page. To see what they see, fetch the page with curl and search the output for the words that matter to you.

Our checklist after every deploy

robots.txt allows the crawlers you want. Under User-agent: * with Allow: /, a crawler that is not named counts as allowed.
Every name in the loop above gets 200.
On Cloudflare, the AI bot policies say the same thing as robots.txt.
Your main words, prices and contact details are in the HTML before any script runs.
Your email address prints in the curl check.
The sitemap lists every page, with dates that change only when a page's words change.

We run these on nicasiadigital.com after every deploy, from a script that lists anything that fails. If you want the same checks run on your site, a private review covers them.

Sources

OpenAI, Overview of OpenAI crawlers
Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
Perplexity, Perplexity crawlers
Google, Google's common crawlers
Cloudflare, press release, 1 July 2025
Cloudflare, Email Address Obfuscation
Vercel and MERJ, The rise of the AI crawler, 17 December 2024