ChatGPT vs Claude for Coding: Task by Task

Search "chatgpt vs claude for coding" right now and every result stops at the same place. playcode.io's top-ranking "real world" task is a debounce function, and it never shows the code, only a narrative summary of each model's result ("functional but used any types in a few places," for one) before pitching its own $9.99/month model bundle against "$60/month for separate subscriptions." That's exactly why we picked harder tasks. developersdigest.tech does show TypeScript snippets, but they're labeled as tendencies, "Claude tends to write this," "GPT tends to write this," illustrative examples rather than the output of a specific run, and never executed. descope.com is the strongest of the set, with real screenshots of both models' actual output, but it's frozen at Claude 3.7 Sonnet against GPT-4o as of 2025-05-30, both superseded generations now, and it analyzes that output without ever executing it, even noting its own generated React code "uses an imaginary backend URL" untested. Everyone else cites a third-party benchmark number and calls that a comparison.

Not one of those pages runs the code. Not one executes a generated file against a test that was written before anyone saw the output. We did that four times, and did two of the four tasks three times each, because a single run tells you what a model can produce and nothing about whether it produces that reliably.

That's the same execution-over-narrative standard Autonoma runs against every pull request in a real codebase, just applied here by hand to a single comparison instead of continuously to a production repo.

How We Ran This Claude vs ChatGPT Coding Test

OpenAI side: gpt-5.6-luna through Codex CLI 0.133.0, invoked with codex exec in a read-only sandbox. Anthropic side: claude-opus-5 through the Claude Code CLI, invoked with claude -p. Both received byte-identical prompts on Node v24.14.0. Single shot, one prompt in, one file out, no tools, no follow-up turns, no iteration on failure.

Acceptance tests were written before any code was generated and never touched afterward. The SHA-256 hashes of every prompt and every test file are committed in results/TESTS_LOCKED.txt, so the tests can't have been adjusted after the fact to fit what either model produced. The only normalization applied to any output was stripping a wrapping markdown fence on runs where a model added one despite being told not to. Claude did this on tasks 1, 2, and 4. GPT never did.

Tasks 1 and 4 were sampled three times per model, specifically to separate stable behavior from run-to-run variance. Tasks 2 and 3 were sampled once, and both models passed everything on that single run.

Every cell is the result of executing a generated file against a test suite hashed before generation. Tasks 2 and 3 were sampled once and tied, so their remaining columns are empty rather than failed. The one filled cell is the whole reason this article samples more than once.

Task 1: One Missing Claim, Two Different Status Codes

Both models got the same prompt: a single Express endpoint, Bearer JWT verified with HS256, a required analyst role, and exact status codes for every failure path, 401 for anything wrong with the token itself, 403 for a valid token that just lacks the role.

Here's the exact prompt both models saw:

Write a single CommonJS file that exports a function `createApp()` returning an Express 4 application.

The app must expose exactly one endpoint: `GET /api/reports/:id`

Requirements:
- Authentication is a Bearer JWT in the `Authorization` header, formatted exactly as `Authorization: Bearer <token>`.
- Verify the token with HS256 using the secret in `process.env.JWT_SECRET`.
- Respond `401` with JSON body `{"error":"unauthorized"}` if the header is missing, malformed, or the token is invalid, expired, or signed with the wrong key.
- The decoded payload contains `sub` (a string user id) and `roles` (an array of strings).
- The endpoint requires the role `analyst`. If the token is valid but the caller does not have that role, respond `403` with JSON body `{"error":"forbidden"}`.
- On success respond `200` with JSON body `{"id": "<the :id route param>", "ownerId": "<payload.sub>"}`.

Output ONLY the raw contents of the file. No markdown fences, no commentary, no explanation. The first characters of your reply must be the first characters of the file.

GPT scored 7 out of 8 on all three runs, failing the same assertion every time: a valid, correctly signed token whose payload has no roles claim at all. GPT returns 401. Claude scored 8 out of 8 on all three runs, returning 403 for the same case. Here's the file GPT produced, where the roles-array check lives inside the same guard as the signature check:

const express = require('express');
const jwt = require('jsonwebtoken');

function createApp() {
  const app = express();

  app.get('/api/reports/:id', (req, res) => {
    const authorization = req.get('Authorization');
    const match = /^Bearer ([^\s]+)$/.exec(authorization || '');

    if (!match) {
      return res.status(401).json({ error: 'unauthorized' });
    }

    let payload;

    try {
      payload = jwt.verify(match[1], process.env.JWT_SECRET, {
        algorithms: ['HS256'],
      });
    } catch {
      return res.status(401).json({ error: 'unauthorized' });
    }

    if (
      !payload ||
      typeof payload !== 'object' ||
      typeof payload.sub !== 'string' ||
      !Array.isArray(payload.roles) ||
      payload.roles.some((role) => typeof role !== 'string')
    ) {
      return res.status(401).json({ error: 'unauthorized' });
    }

    if (!payload.roles.includes('analyst')) {
      return res.status(403).json({ error: 'forbidden' });
    }

    return res.status(200).json({
      id: req.params.id,
      ownerId: payload.sub,
    });
  });

  return app;
}

module.exports = { createApp };

And here's Claude's version, which splits authentication and authorization into two separate middleware functions:

'use strict';

const express = require('express');
const jwt = require('jsonwebtoken');

const REQUIRED_ROLE = 'analyst';

function unauthorized(res) {
  return res.status(401).json({ error: 'unauthorized' });
}

function authenticate(req, res, next) {
  const header = req.headers.authorization;

  if (typeof header !== 'string') {
    return unauthorized(res);
  }

  const parts = header.split(' ');

  if (parts.length !== 2 || parts[0] !== 'Bearer' || parts[1].length === 0) {
    return unauthorized(res);
  }

  const secret = process.env.JWT_SECRET;

  if (typeof secret !== 'string' || secret.length === 0) {
    return unauthorized(res);
  }

  let payload;
  try {
    payload = jwt.verify(parts[1], secret, { algorithms: ['HS256'] });
  } catch (err) {
    return unauthorized(res);
  }

  if (payload === null || typeof payload !== 'object') {
    return unauthorized(res);
  }

  if (typeof payload.sub !== 'string' || payload.sub.length === 0) {
    return unauthorized(res);
  }

  req.auth = payload;
  return next();
}

function requireRole(role) {
  return function (req, res, next) {
    const roles = req.auth && req.auth.roles;

    if (!Array.isArray(roles) || !roles.includes(role)) {
      return res.status(403).json({ error: 'forbidden' });
    }

    return next();
  };
}

function createApp() {
  const app = express();

  app.disable('x-powered-by');

  app.get(
    '/api/reports/:id',
    authenticate,
    requireRole(REQUIRED_ROLE),
    function (req, res) {
      res.status(200).json({ id: req.params.id, ownerId: req.auth.sub });
    }
  );

  return app;
}

module.exports = { createApp };
module.exports.createApp = createApp;

The spec never says what a missing roles claim means. GPT treats a payload with no roles array as malformed and rejects it at the authentication layer, before authorization ever runs. Claude authenticates on sub alone, then checks the role separately, and a payload with no roles simply fails that separate check with 403. Both are defensible readings of an underspecified requirement. Neither crashed: the naive failure mode here is a 500 from calling .includes on undefined, and both models guarded against it. The point isn't that one model is wrong. It's that the difference is stable across three identical runs each, and it's invisible unless you execute both files against the same edge case. Catching that kind of edge case by hand, for every pull request, doesn't scale, which is the specific job Autonoma automates: generating the behavioral check once and running it against your application continuously, instead of once, by hand, for a single article.

Tasks 2 and 3: Both Tied, Nothing to Report

Task 2 was a React component with out-of-order async responses layered on a stale-closure bug, 4 assertions. Task 3 was a callback-style retry module refactored into three ES modules with async/await, preserving retry and error semantics exactly, 7 assertions. GPT passed both cleanly on the single sampled run. Claude passed both cleanly on the single sampled run. No divergence, no story.

Passing task 2 meant a slower response for an earlier keystroke could not overwrite a later one's results, and clearing the input mid-flight had to leave the list empty. Both models solved it the same way: a boolean flag set on cleanup that blocks a resolved response from updating state. Claude's version also attaches a no-op rejection handler to the search promise, avoiding an unhandled rejection on a failed fetch; GPT's doesn't. The generated files (generated/claude/task2/SearchBox.jsx and generated/gpt/task2/SearchBox.jsx) sit in the repo if you want to see the full diff, but a tie is a tie. A tie is exactly the case where task type stops driving the choice, and it's the same split covered in more depth in picking a model per coding task.

Task 4: The Run Where Seven Records Went Missing

This is the one worth the space. The prompt asked for a keyset pagination helper as an ES module, ordering by createdAt descending with ties broken by id, walking the full result set exactly once with no skips and no duplicates across a page boundary that happens to land on a tie:

Implement a keyset pagination helper as an ES module.

Export a single function:

```js
export function paginate(items, options)
```

- `items` is an array of objects shaped `{ id: string, createdAt: number, title: string }`. Treat it as an unordered result set; do not assume it arrives sorted, and do not assume it arrives in the same order on every call.
- `options` is `{ limit, cursor }`. `limit` is a positive integer. `cursor` is either `null`/`undefined` for the first page, or a value previously returned as `nextCursor`.
- Ordering is `createdAt` descending, with ties broken by `id` ascending using ordinary string comparison.
- Return `{ rows, nextCursor }`. `rows` contains at most `limit` items in that order. `nextCursor` is an opaque string used to fetch the following page, or `null` when the returned page is the last one.
- Repeatedly calling `paginate`, feeding the previous `nextCursor` back in, must walk the entire result set in the specified order and return every item exactly once. No item may be skipped and no item may appear on two pages.
- `nextCursor` must be an opaque string. It must not be a numeric offset or an index into the input array, and the helper must not depend on the positional order of the input array.
- An empty `items` array returns `{ rows: [], nextCursor: null }`.

Output ONLY the raw contents of the module file. No markdown fences, no commentary, no explanation. The first characters of your reply must be the first characters of the file.

GPT passed 7 out of 7 on all three runs. Claude passed 4 out of 7, then 7 out of 7, then 7 out of 7. On that first run, Claude's paginator returned page 1 correctly, five rows with a valid cursor, then returned an empty page 2 with nextCursor: null. No error, no crash, no warning. We reproduced it directly rather than trusting only the test output: page 1 came back with rows a1, b2, c3, d4, e5 and a cursor. Page 2, fed that exact cursor, came back with zero rows and a null cursor. Seven of the twelve input records became silently unreachable.

Both panels describe the same generated file. Nothing on the left is wrong, which is exactly the problem: sophistication is not correctness, and only the right-hand panel required running the code.

Here's the file that produced that run:

const CURSOR_PREFIX = 'kc1.';

function compare(a, b) {
  if (a.createdAt !== b.createdAt) return b.createdAt - a.createdAt;
  if (a.id < b.id) return -1;
  if (a.id > b.id) return 1;
  return 0;
}

function encodeCursor(item) {
  const payload = JSON.stringify({ c: item.createdAt, i: item.id });
  const bytes =
    typeof TextEncoder !== 'undefined'
      ? new TextEncoder().encode(payload)
      : null;
  let binary = '';
  if (bytes) {
    for (let i = 0; i < bytes.length; i += 1) {
      binary += String.fromCharCode(bytes[i]);
    }
  } else {
    binary = unescape(encodeURIComponent(payload));
  }
  const base64 =
    typeof btoa === 'function'
      ? btoa(binary)
      : Buffer.from(payload, 'utf8').toString('base64');
  return CURSOR_PREFIX + base64.replace(/\+/g, '-').replace(/\//g, '_').replace(/=+$/, '');
}

function decodeCursor(cursor) {
  if (typeof cursor !== 'string' || !cursor.startsWith(CURSOR_PREFIX)) {
    throw new TypeError('Invalid cursor');
  }
  let base64 = cursor.slice(CURSOR_PREFIX.length).replace(/-/g, '+').replace(/_/g, '/');
  while (base64.length % 4 !== 0) base64 += '=';

  let payload;
  if (typeof atob === 'function') {
    const binary = atob(base64);
    const bytes = new Uint8Array(binary.length);
    for (let i = 0; i < binary.length; i += 1) bytes[i] = binary.charCodeAt(i);
    payload =
      typeof TextDecoder !== 'undefined'
        ? new TextDecoder().decode(bytes)
        : decodeURIComponent(escape(binary));
  } else {
    payload = Buffer.from(base64, 'base64').toString('utf8');
  }

  let parsed;
  try {
    parsed = JSON.parse(payload);
  } catch {
    throw new TypeError('Invalid cursor');
  }
  if (
    parsed === null ||
    typeof parsed !== 'object' ||
    typeof parsed.c !== 'number' ||
    typeof parsed.i !== 'string'
  ) {
    throw new TypeError('Invalid cursor');
  }
  return { createdAt: parsed.c, id: parsed.i };
}

export function paginate(items, options) {
  const { limit, cursor } = options ?? {};

  if (!Number.isInteger(limit) || limit <= 0) {
    throw new TypeError('limit must be a positive integer');
  }
  if (!Array.isArray(items) || items.length === 0) {
    return { rows: [], nextCursor: null };
  }

  const sorted = items.slice().sort(compare);

  let start = 0;
  if (cursor !== null && cursor !== undefined) {
    const after = decodeCursor(cursor);
    // Binary search for the first item strictly after the cursor position.
    let lo = 0;
    let hi = sorted.length;
    while (lo < hi) {
      const mid = (lo + hi) >> 1;
      if (compare(sorted[mid], after) <= 0) hi = mid;
      else lo = mid + 1;
    }
    start = lo;
    // The cursor item itself, if still present, must be excluded.
    if (
      start < sorted.length &&
      sorted[start].createdAt === after.createdAt &&
      sorted[start].id === after.id
    ) {
      start += 1;
    }
  }

  const rows = sorted.slice(start, start + limit);
  const hasMore = start + rows.length < sorted.length;

  return {
    rows,
    nextCursor: hasMore && rows.length > 0 ? encodeCursor(rows[rows.length - 1]) : null,
  };
}

The bug is in the binary search that locates the cursor position. It branches hi = mid when compare(sorted[mid], after) <= 0, which is the correct shape for searching a sequence that goes from false to true, but the actual sequence here goes from true to false, since the list is sorted descending and the search is looking for the first item strictly after the cursor. The search walks off the end, start lands at 12 in a 12-item array, and slice(12, 12 + limit) returns nothing.

For contrast, here's GPT's version of the same file, which passed on all three runs:

function encodeCursor(createdAt, id) {
  return Buffer.from(JSON.stringify({ createdAt, id }), "utf8").toString(
    "base64url",
  );
}

function decodeCursor(cursor) {
  const decoded = Buffer.from(cursor, "base64url").toString("utf8");
  const value = JSON.parse(decoded);

  if (
    !value ||
    typeof value !== "object" ||
    typeof value.createdAt !== "number" ||
    typeof value.id !== "string"
  ) {
    throw new TypeError("Invalid cursor");
  }

  return value;
}

export function paginate(items, options) {
  const { limit, cursor } = options;

  if (!Number.isInteger(limit) || limit <= 0) {
    throw new TypeError("limit must be a positive integer");
  }

  if (items.length === 0) {
    return { rows: [], nextCursor: null };
  }

  const orderedItems = [...items].sort((left, right) => {
    if (left.createdAt !== right.createdAt) {
      return right.createdAt - left.createdAt;
    }

    return left.id < right.id ? -1 : left.id > right.id ? 1 : 0;
  });

  const boundary = cursor == null ? null : decodeCursor(cursor);

  const startIndex =
    boundary == null
      ? 0
      : orderedItems.findIndex(
          (item) =>
            item.createdAt < boundary.createdAt ||
            (item.createdAt === boundary.createdAt && item.id > boundary.id),
        );

  const rows = orderedItems.slice(
    startIndex === -1 ? orderedItems.length : startIndex,
    (startIndex === -1 ? orderedItems.length : startIndex) + limit,
  );

  const nextItemIndex =
    (startIndex === -1 ? orderedItems.length : startIndex) + rows.length;

  return {
    rows,
    nextCursor:
      nextItemIndex < orderedItems.length
        ? encodeCursor(
            rows[rows.length - 1].createdAt,
            rows[rows.length - 1].id,
          )
        : null,
  };
}

Say the uncomfortable part plainly: the failing file is arguably the better-engineered one. It has base64url cursor encoding with environment fallbacks for btoa/Buffer, cursor payload validation that throws on tampering, a correct tie-breaking comparator, and an explicit, documented guard to exclude the cursor item itself. GPT's passing version is a plain findIndex linear scan with none of that scaffolding. A reviewer comparing the two files on sophistication alone would plausibly prefer the one that silently loses data.

Per-Task Verdict: ChatGPT vs Claude for Coding

Across the four tasks, ChatGPT vs Claude for coding did not resolve to a single winner. Here is how each one landed.

Same Model, Same Prompt: Two Outcomes for Claude vs GPT Coding

A single-shot comparison, which is exactly what every ranking article on this query already is, would have run task 4 once and concluded "Claude is broken at cursor pagination." That conclusion would have been wrong two-thirds of the time on this exact evidence. It also would have missed the task 1 divergence on 401 versus 403 entirely, since that difference is deterministic and shows up regardless of which run you happen to sample. The useful question isn't which model writes better code in the abstract. It's what you do about the fact that the same model, given the same prompt, can hand you the working file or the one that quietly drops most of your dataset, and you can't tell which one you got by reading it.

That's the same gap every page ranking for this query shares, including the general-verdict version of this comparison at Claude vs ChatGPT for coding: none of them execute anything. Autonoma runs those same behavioral checks against the actual running feature on every pull request, continuously, because what caught the pagination bug in this article was executing the code at all, and doing that by hand, once, for a single comparison is exactly what doesn't scale.

What This Doesn't Prove

Three samples on two tasks and one sample on the other two is a small n. This is an existence proof that the variance exists, not a measured failure rate for either model. Anyone quoting a percentage off this specific dataset is doing exactly the thing this article is arguing against. Both vendors ship several models at several tiers; these are two specific ones, run once and three times respectively, on one day. A different prompt, a different task, or a different tier could easily flip which model looks more consistent.

None of this makes one model a categorically safer default than the other. What it argues for is checking the code that actually got generated, every run, not just the run someone happened to sample before publishing a verdict. If the changelog of what's shifted between these two models this year matters more than a single test run, what changed between ChatGPT and Claude for coding in 2026 covers that. If the real question is which model to reach for on a specific kind of task rather than in general, picking a model per coding task goes further into that split.

It's also the same claim Autonoma is built around: whichever model or agent last touched a codebase, the running application gets verified on the next pull request, not sampled once and assumed to hold.

Frequently Asked Questions