Errors are values, and there is no unwrap

The format has a lot of ways to fail: wrong magic bytes, an unsupported version, a read-only file, a filename with no instability value. I wanted every one of those to be a specific, named error rather than a panic or a bare string. So the whole crate returns a single enum:

pub enum DecayError {
    WrongMagic { found: [u8; 4] },
    UnsupportedVersion { found: u8 },
    ReadOnly { path: String },
    FilenameNoX { filename: String },
    // ...and so on, one variant per way things fail
}

Every fallible function returns Result<_, DecayError>, and ? threads failures up to main, which is the only place anything is printed. I deliberately did not reach for thiserror; there were few enough variants that hand-writing the Display impl read more clearly than a macro and kept the dependency list short. The rule I held was: no unwrap() outside tests.

What I took from this: making errors values rather than panics forced me to name every failure up front instead of discovering them at runtime. By the time the enum was written, I had basically enumerated the entire contract of how the format can refuse you, which turned out to be a good way to think about the format itself.

Parsing untrusted bytes without panicking

A decayfmt file starts with a fixed 16-byte header. The lazy way to read it is to index straight into the slice, but indexing panics if the buffer is short, and a truncated or tampered file is exactly the case you have to survive. So read checks the length first, validates in order, and only hands back a header if everything matches:

pub fn read(buffer: &[u8]) -> Result<Header, DecayError> {
    if buffer.len() < HEADER_SIZE {
        return Err(DecayError::PayloadTooSmall {
            found: buffer.len(),
            needed: HEADER_SIZE,
        });
    }

    let mut found_magic = [0u8; 4];
    found_magic.copy_from_slice(&buffer[0..4]);
    if found_magic != MAGIC {
        return Err(DecayError::WrongMagic { found: found_magic });
    }
    // version and file_type validated the same way, each a typed refusal
}

The little-endian image dimensions are read with u32::from_le_bytes on a fixed four-byte array, not manual bit shifting.

What I learned: the habit of length-check-then-read, so that a malformed file is always a typed refusal and never a panic. from_le_bytes was new to me too, and it is a lot harder to get wrong than shifting bytes by hand.

The randomness is never seeded

Corruption picks bytes to replace with a probability derived from the instability value:

fn corrupt_text<R: Rng>(payload: &mut [u8], probability: f64, rng: &mut R) {
    for byte in payload.iter_mut() {
        if rng.gen::<f64>() < probability {
            *byte = rng.gen_range(0x20..=0x7E); // a random printable ASCII byte
        }
    }
}

The generator is rand::thread_rng(), a cryptographically secure generator seeded from OS entropy, and it is never seeded with a fixed value. That is the one genuinely security-relevant decision in the project: a seeded, reproducible RNG would make the corruption sequence replayable, and a replayable corruption is a recoverable one, which defeats the whole point of the format. Making the function generic over the Rng trait (<R: Rng>) had a nice side effect: a test can hand it any generator it likes, including a seeded one, without the function knowing or caring.

What I learned: the difference between thread_rng and a SeedableRng, and that “generic over a trait” is not just an abstraction for its own sake here; it is the seam that lets a test control the randomness when it needs to.

Writing decay to disk without bricking the file (the one I got wrong first)

This is the part I had to fix after the first release. My initial version wrote the corrupted file back with std::fs::write:

std::fs::write(path, &file_bytes)?; // rewrites the whole file, header included

That works until it doesn’t. fs::write truncates the file and rewrites it from scratch, header and all. If the process is killed mid-write, you can be left with a zero-length or half-written file whose header is gone, which means it can never be opened again. For a format whose entire promise is “the file decays,” turning it into an unopenable brick on a bad crash is the wrong failure mode. It also quietly rewrote the header on every open, which the format contract says must never happen.

The fix was to open the file, seek past the header, and overwrite only the payload region in place:

let mut file = OpenOptions::new().write(true).open(path)?;
file.seek(SeekFrom::Start(HEADER_SIZE as u64))?;
file.write_all(&file_bytes[HEADER_SIZE..])?;

Because corruption never changes the payload’s length, this overwrites the same number of bytes with no truncation. The header on disk is now literally never touched, and a crash mid-write leaves a partially corrupted payload behind an intact header. The file still decays; it just cannot brick.

What I learned, and this was the real lesson of the project: the convenient standard-library call is not always the correct one. fs::write is perfect when you are writing a whole new file and terrible when you need to surgically overwrite part of an existing one. Reaching for OpenOptions, Seek, and Write instead was the difference between a durable format and a fragile one.

Testing something that is different every run

Testing corruption is strange because the output changes every time. You cannot assert on exact bytes. What you can assert on is statistical properties. At an instability of 1 the per-byte probability is about 0.095, so over a large buffer roughly 9.5% of bytes should change:

let original = vec![0u8; 100_000];
let mut payload = original.clone();
corrupt(&mut payload, FileType::Text, 1.0);
let fraction = changed_count(&original, &payload) as f64 / original.len() as f64;
assert!((fraction - 0.095).abs() < 0.01);

The “alpha channel is never corrupted” invariant is tested the same way: corrupt a known image payload many times over and confirm every alpha byte survived untouched. These live in inline #[cfg(test)] modules and run with cargo test; CI also runs cargo clippy -- -D warnings and cargo fmt --check, so nothing merges unformatted or with a warning.

What I learned: how to test randomness without pretending it is deterministic. You assert on the shape of the distribution over a big enough sample, not on any single output, and you use a large buffer so the law of large numbers does the work for you.

Closing

None of this is advanced Rust. But writing a format that has to be honest about failure, refuse bad input with a typed error, never hand back a free read, and never brick on a crash, pushed me toward exactly the parts of the language that make that kind of correctness cheap: enums for errors, ? for propagation, traits for testable randomness, and the standard library’s precise file handles for when the convenient one-liner is not enough.

The whole thing is a few hundred lines and MIT-licensed if you want to read it: github.com/aravpanwar/decayfmt.

On 12 June 2026, a group calling itself World Leaks posted more than 630GB of data to its dark-web leak site: somewhere north of 204,000 files. Most of it traced back, one way or another, to Apple.

Tata Electronics confirmed a “cybersecurity incident” on some of its systems and said operations and manufacturing were unaffected. Both halves of that are true, and the second half describes the part nobody was worried about. The factories kept assembling phones. What walked out of the building was the paperwork explaining how the phones get made.

According to the reporting, the cache included supplier-mapping documents for the unreleased iPhone 18 Pro and Pro Max: hundreds of components traced back to specific vendors, which suppliers were competing for which contracts, and, for good measure, photographs from drop-test prototypes. Files touching Tesla, TSMC and Qualcomm were in there too, though the bulk of it centred on Apple. Apple has said it is concerned and investigating, and that its security team is working with Tata.

One caveat to nail down before going further, because it stays attached to everything that follows. Reuters and TechCrunch both noted they could not independently verify the authenticity, origin or completeness of the files. So every specific detail drawn from the cache is reported, not confirmed. This matters less than you would expect, because the interesting part of this story is not what was in the files. It is where they were.

the group

World Leaks does not encrypt anything. It steals data and threatens to publish it, and if you do not pay, it publishes. The group is a 2025 rebrand of Hunters International, which spent 2023 and 2024 as a conventional ransomware operation before dropping the encryption step under law-enforcement pressure. This is worth sitting with for a second. Encrypting a victim’s files is the hard, loud, arrestable part of ransomware. Threatening to leak them is the easy part. Hunters International looked at that trade and decided to do less work, and the reporting suggests the decision has been good for business.

How they got into Tata is not known. Tata has hired a global consultant for a forensic audit, restricted remote and external access to sensitive systems such as purchase-order processing, reported the incident to CERT-In and to its clients, and notified some of its iPhone-assembly staff. A ransom was demanded. As of the end of June the investigation was still described as ongoing, which means anyone stating the root cause with confidence right now is guessing.

The most-quoted guess belongs to Paolo Pescatore of PP Foresight, who told Al Jazeera that a breach of this size is “not usually a smash-and-grab,” and that moving this much data typically requires a foothold, compromised credentials, weak access controls or undetected lateral movement. The load-bearing word there is typically. It describes how breaches of this kind generally happen, not what happened here; the forensic findings will decide that. Pescatore then added the sentence that carries the rest of this piece: the foothold need not be within Apple itself, but within a supplier.

the story is the supply chain, not Tata

That is the whole thing. Apple’s product secrets left through a vendor, and no Apple system had to be touched for it to happen. You can run world-class security at the centre of a network and still lose the crown jewels through a node two hops away that you do not control, do not audit continuously, and cannot patch on your own schedule.

The reason this is worth writing about, rather than filing under bad-luck-of-the-week, is the speed of the thing it sits on top of. India is on track to assemble around 26% of the world’s iPhones in 2026, up from roughly 6% four years earlier (Counterpoint Research, via Reuters). Tata accounts for about a third of Apple’s India output; Foxconn makes the rest. And Tata got to that third quickly, by acquisition rather than by building: it bought Wistron’s India iPhone plant in October 2023 for about $125M, then took a 60% controlling stake in Pegatron’s India operations, completed in January 2025. That leaves it running three iPhone factories and more than 75,000 employees, second only to Foxconn among Apple’s India suppliers. Tata assembled its position in the supply chain roughly the way it assembles the phones: by buying the components and putting them together at speed.

The temptation now is to point at one incident and call it a trend. That does not work on a single data point, so here are three, with a warning label on the first.

The warning label: in December 2020, workers at Wistron’s Narasapura plant, the same plant Tata would later buy, ransacked it over unpaid wages, causing around $7M in damage. A Karnataka state inquiry found serious labour-law violations and concluded, in effect, that Wistron could not cope with the speed of its own scale-up. That is a real and relevant pattern, but it is a labour and physical-controls failure, not a data breach. Its value here is the theme, a supplier whose controls did not keep pace with how fast Apple grew it, and nothing more. Anyone using it as a cyber precedent is quietly switching the ball for a different ball.

The second point is a cyber one, and it is the spine of the argument. Jaguar Land Rover, owned by Tata Motors, was breached twice in 2025. The first, in March, was reportedly enabled by a third party’s stolen credentials that had access to JLR’s Jira, which is to say a supply-chain vector, and it leaked hundreds of gigabytes. The second, on 31 August, halted JLR’s global production for weeks and has been described as the most damaging cyberattack in British history, with an economic impact near £1.9B; Tata Motors itself disclosed costs of around £1.8B, about $2.35B, including £196M in exceptional charges. The security analysis points to credential abuse, lateral movement and weak network segmentation.

Now line the two up. The tactics analysts infer for Tata Electronics, compromised credentials plus lateral movement, are the tactics confirmed at another Tata-owned manufacturer, in the same year, under the same parent company. That is not a parallel you have to strain to draw. It is on the record.

The third point is just the weather. Manufacturing was the most-attacked ransomware sector in the world in 2025, and India was the loudest corner of it. The Tata Electronics leak is not an outlier against that backdrop. It is a data point on-trend.

the regulatory floor, and what it does not cover

Here is where the obvious framing is slightly wrong, so it is worth slowing down.

India’s flagship data law is the Digital Personal Data Protection Act. The word doing the quiet work in the title is personal. The Act governs digital personal data of individuals. The crown jewels in this breach, supplier maps, component-to-vendor linkages, prototype photographs, are corporate and commercial intellectual property, not personal data, and the reporting indicates no consumer or user data was taken. So the honest version of the regulatory story is not “the DPDP would have prevented this.” For most of what was actually lost, the DPDP is the wrong instrument. The things meant to protect supplier and prototype secrets are trade-secret law, contracts, and incident-reporting rules, not a privacy statute.

What the DPDP is, is the closest thing India has to a GDPR-style forcing function for vendor security, and its immaturity is a fair emblem of a wider compliance floor that is still being poured. So the claim worth making is about the floor, not about this particular file set.

The floor, as of mid-2026, looks like this. The Act itself is from 2023. Its rules were only notified by the government in November 2025, and they arrive with a phased, roughly eighteen-month transition, with full compliance expected around May 2027. The Data Protection Board of India was stood up on notification. The breach-notification rule, when it bites, is genuinely strict: a two-stage process, an immediate initial intimation and then a detailed report within 72 hours, with no minimum threshold, so a one-person breach and a 204,000-file breach carry the same duty to report. The headline penalties run up to ₹250 crore for failing to implement reasonable security safeguards and up to ₹200 crore for failing to report a breach, which is about $25 to $30M and, on paper, in GDPR’s weight class.

On paper is carrying that last sentence, because at the moment World Leaks posted the files, those penalties were not yet enforceable. Until the DPDP’s core provisions take effect around May 2027, the in-force regime is the older IT Act and the CERT-In directions, whose binding obligation is a six-hour incident-reporting rule from April 2022. Tata’s reported action, notifying CERT-In, is compliance with that rule. It is a rule about telling the authorities quickly. It is not a rule about making the breach less likely, and it is not the ₹250-crore safeguards penalty, which was not yet switched on.

The contrast that makes this legible is GDPR, enforceable since May 2018, so roughly eight years of accumulated pressure and, by the DLA Piper survey, more than €7.1B in cumulative fines, €1.2B of it in 2025 alone. The mechanism that actually reshaped vendor security lives in Article 28: you may use only processors that offer sufficient guarantees, the relationship must run through a binding data-processing agreement, you owe due diligence before you appoint them and audit rights for as long as you use them, sub-processors inherit the same obligations, and liability follows the chain rather than stopping at the first vendor. That is the part with no fully in-force Indian analogue yet. Over a decade, it is what turned “vendor security” from a line item in procurement into something contractual, audited, continuous and liability-bearing for anyone touching EU personal data.

So the defensible shape of the argument is narrow and, I think, true. GDPR spent about eight years and several billion euros converting vendor security into an enforceable duty. India’s equivalent forcing function exists on paper, with comparable headline numbers, but does not fully bite until 2027, and even then points at personal data, which is not where this breach landed. The floor is being poured while the building is already several storeys up.

why the sector is soft

The reason manufacturing keeps turning up at the top of these tables is not that manufacturers are careless. It is structural, and some of the structure is measurable.

The measurable part first, and all of it from one vendor’s report, so read it as Check Point’s count rather than a multi-source consensus. Manufacturing was the most-attacked ransomware sector globally in 2025; Check Point counted a 56% rise in attacks on the sector, from 937 to 1,466, roughly half of all global ransomware activity, and other trackers put the surge higher still. India was the sharp end of the same report: about 65% of affected Indian organisations paid a ransom, at an average payout of $1.35M, and India recorded the second-highest manufacturing-ransomware incident count after the United States. Over one six-month window, Check Point put the figure at up to 2,786 attacks a week on Indian industrial-manufacturing organisations, which is less a threat landscape than a climate.

The reason the sector absorbs this is legacy. Programmable controllers, SCADA systems and industrial IoT were designed in an era when the security model was a locked door and an air gap, and they were never meant to sit on a modern network. As an installed-base analogue, something like 80% of European manufacturers still run critical operational technology with known vulnerabilities, and compromised industrial credentials trade for anywhere from $4,000 to $70,000 on the relevant markets. When IT and OT networks converged, they brought decades of what Forrester’s Paddy Harrington calls inherent trust: the internal doors were left open because for years nothing outside could reach them. Once an attacker crosses from IT into OT, that trust is the thing that lets them wander. This is “lateral movement” described without the jargon.

You can see the maturity gap in how long incidents last. Dragos found that organisations with real visibility into their OT contained incidents in about five days, against an industry average of forty-two. That thirty-seven-day gap is not a gap in available tools; the tools exist and can be bought. It is a gap in whether anyone was watching.

Two further mechanisms usually get named at this point, and I would rather be honest than tidy about them. One is that security in these firms tends to report up through IT rather than holding its own budget and authority. The other is that procurement cycles do not price it in. Both are plausible, and both are, as far as I could find, analyst folklore rather than things I can attach an Indian-manufacturing number to. There is adjacent evidence, but no clean survey saying “in X% of Indian manufacturers, security reports to the CIO with no independent budget.” So take those two as mechanism, not as measurement, until someone puts a figure on them.

The cleanest way to see that the missing ingredient is external pressure rather than internal virtue is to look at the Indian sector that had the pressure. Banking, financial services and insurance is India’s most mature, highest-spend security sector, around 23.88% of the country’s cybersecurity market revenue in 2025, the acknowledged bellwether, operating under a regulator that insists on things like immutable audit trails. It got that way because it has had a regulatory forcing function for over a decade, which is precisely what manufacturing has not had. And here is the part that keeps the story honest: even BFSI, with all of that pressure, saw incidents climb from about 1.4 million to 2.9 million between 2021 and 2025, and, on the BCG and DSCI figures, India’s mean time to contain a breach is 263 days and rising. So the lesson is not “be more like the banks and you will be safe.” It is narrower and bleaker: the banks had a forcing function and are still strained; manufacturing has no forcing function and is scaling faster. Underneath all of it, NASSCOM has estimated a cybersecurity talent shortfall in India running into the millions, which is the sort of number that does not get fixed inside a hiring cycle.

why this is not the last one

Put the three pieces together and the forecast writes itself, except that it is not really a forecast.

A manufacturing base scaling from 6% to 26% of the world’s iPhones in four years, with one supplier consolidating Wistron and Pegatron into a third of Apple’s India output. A compliance floor whose vendor and breach provisions do not bite until 2027 and do not reach commercial data even then, sitting on top of an in-force rule that requires you to report a breach within six hours but does nothing to make one less likely. And a sector carrying decades of legacy security debt, where crossing from IT into OT opens doors that were left unlocked on the assumption nobody could reach them. Those three together do not describe an unlucky event. They describe a standing condition.

The reason it does not need to be argued as a prediction is that the recurrence has already happened, inside the same company. Jaguar Land Rover, under Tata Motors, was breached twice in one year, once through a third party’s credentials, using the exact pattern of credential abuse and lateral movement that analysts merely infer for Tata Electronics. The cybersecurity researcher Rajshekhar Rajaharia named JLR as the precedent and warned of copycats before the Tata Electronics story had finished being written. When the same parent company is hit repeatedly through the same class of weakness while its footprint expands, “it will happen again” stops being a prediction and becomes a description of the trend line.

the part I will leave where it is

Almost all of the security posture in play here, at Tata, at Apple, across the sector, rests on one premise: that attackers can be kept out. Perimeters and detection are bets on prevention.

A supply chain expanding at this speed multiplies the number of vendor nodes faster than anyone can harden them, and each node is a separate, independent way in. The premise does not have to fail everywhere. It has to hold everywhere, at every node, every day, and the attacker only needs it to fail once.

A boom that adds nodes faster than it can secure them is, in the end, a bet that a premise which only has to break once will keep on not breaking. So far, that bet is being placed several storeys up, on a floor that is still being poured.

links and sources

Verified through 1 July 2026. The forensic investigation was still described as ongoing, with no confirmed root cause and no confirmed ransom outcome, so the incident section reflects reporting as it stood, not a settled account.

The incident. Reuters’ reporting is the origin for almost every specific detail; both Reuters and TechCrunch noted they could not independently verify the authenticity, origin or completeness of the files, so the leak-derived specifics are reported, not confirmed.

• Al Jazeera, Apple iPhone 18 Pro secrets leaked in Tata Electronics hack: what we know (Pescatore and Rajaharia quotes): aljazeera.com
• Storyboard18, Apple supplier Tata Electronics steps up cybersecurity after alleged leak (forensic audit, CERT-In, Counterpoint 26%/6%): storyboard18.com
• Quartz, Apple iPhone 18 Pro supplier list leaked in Tata data leak (Reuters-reviewed component-to-vendor detail): qz.com
• Reuters via Yahoo, Apple iPhone 18 Pro secrets leaked in Tata Electronics hack: yahoo.com

The pattern (JLR). The spine of the argument. The £1.9B economy-wide figure is the Cyber Monitoring Centre’s estimate; the ~£1.8B / $2.35B cost and the £196M exceptional charge are Tata Motors’ own disclosure.

• Jaguar Land Rover cyberattack, overview and £1.9B figure: en.wikipedia.org
• CNBC, JLR cyberattack holds an ominous lesson for British businesses (Tata/TCS outsourcing, CMC estimate): cnbc.com
• The Register via Tech Digest, JLR hack cost Tata Motors ~£1.8B (£196M exceptional charge): techdigest.tv
• Business Today, JLR hack, UK’s worst cyber event yet: businesstoday.in

The regulatory floor (DPDP and GDPR). DPDP Rules were notified 13 to 14 November 2025; the ₹250-crore and ₹200-crore ceilings are statutory maximums whose enforcement begins only after the ~May 2027 deadline. A January 2026 MeitY consultation floated compressing the transition to twelve months, but that was not gazetted, so ~May 2027 remains the baseline.

• EY India, DPDP Rules 2025 notified by MeitY: complete guide (72-hour report, phased 18-month rollout): ey.com
• Seclore, DPDP Rules 2025 compliance guide (₹250cr / ₹200cr penalties, Rule 7): seclore.com
• ConsentOS, DPDP breach-notification timeline (two-stage, no threshold, CERT-In 6-hour dual clock): consentos.in
• S.S. Rana & Co., MeitY notifies final DPDP Rules 2025 (transition dates: Nov 2026, May 13 2027): ssrana.in
• CMS GDPR Enforcement Tracker Report, numbers and figures: cms.law
• Bitdefender, summarising the DLA Piper January 2026 survey (€7.1B cumulative, €1.2B in 2025): bitdefender.com

The sector (why it is soft). The manufacturing and India figures are all from a single vendor report, Check Point’s Manufacturing Threat Landscape 2025, and are attributed as such in the text. The 80% figure is European operational-technology data used as an installed-base analogue, not an India number. The 263-day containment figure comes from a BCG/DSCI report that frames a largely BFSI-sourced number as economy-wide; treat its provenance accordingly.

• Digital Terminal, summarising Check Point’s Manufacturing Threat Landscape 2025 (56% rise, 65% paid, $1.35M, 2,786/week): digitalterminal.in
• Check Point Research, The State of Ransomware Q3 2025 (manufacturing as top sector): research.checkpoint.com

original : This sentence is dying, and every time you read it you kill it a little more.
 open 1  : This sgntence is d+ingd !nd every time you re&p it P~u kiKl it a little more}
 open 3  : This sgfxFn0e is d+ingd 3D6 every tibe you re&" it P~u kiKl it a 1ittl> m1re}
 open 6  : TIbm sgf}Fn0e ts d+iqgd yD6 ev*ry tibe you re&" )t Pnu kiKB )t aC1it"l> m1^e}
 open 9  : T/Sm sgf}Fk0- ts d|iqgd HD6 e@*rV tiFe you re&" )t Pnu kiKB )tpaC1it"lYMm1^>}
 open 12 : h/Sm hgf}Nk0-'ts?K|iqgd HD6 e@`~V}t&Fe y%u re&" )2 Pnu kiKB )6UaC1it1lYMm1]b!

That is one sentence in a text file, opened a few times. Nothing edited it between reads. Each open is the only thing that touched it, and each one corrupted it a little more, on disk, permanently, before it showed me anything. The first line is gone now. What is above is all that is left of it, and there is no way back to it.

The tool is called decayfmt. You encode an image or a text file into it, and after that, every time you open it, it permanently corrupts a little before showing you anything. The corruption happens at open time and is written to disk before the file is displayed, not when you close it and not later in the background. If the program is killed right after that write, the damage is already on disk and stays there. There is no recovery from the file alone. The only copy is the one you keep degrading by looking at it.

How quickly it degrades is set by a number in the filename. Images are named like photo.idcy3 and text like note.tdcy7, and the trailing number is the instability, higher meaning more corruption per open. That number is read from the filename and nowhere else, so renaming a file changes how hard the next open hits it. I kept it in the name rather than buried in the header mostly because it is hard to forget it is there.

The corruption is a per-byte chance derived from that number. The probability that any given byte is replaced on a single open is 1 - exp(-x / 10), so an instability of 1 corrupts around a tenth of the eligible bytes each time and 10 corrupts around two thirds, which is to say it is nearly gone after one look. The choice of which bytes, and what to replace them with, comes from the operating system’s cryptographic random generator and is never seeded. That is the one place I was careful. A seeded generator would make the sequence reproducible, and a reproducible corruption sequence is a recoverable one, which would undo the entire point.

There is one behaviour that looks like a bug and is not. If the file is read-only, decayfmt refuses to open it and returns an error instead of showing you anything. The reasoning is that the format’s single rule is that opening costs a corruption, and a file that cannot be written to cannot be corrupted, so opening it would be a free read. Rather than allow that, it fails closed. A locked decayfmt file is not protected, it is just stuck.

The sentence at the top stays readable as it rots because the bytes it swaps in are restricted to printable characters, so text degrades into something half-remembered instead of binary garbage. Images have nothing keeping them legible. Below is one image encoded four times at four different instability values and opened in step, so the same frame decays at four rates at once. The alpha channel is left alone on purpose, so the damage shows up as colour noise rather than holes punched through the picture.

I thought a file you could use up would be funny, so I built one. It works exactly as badly as intended, and I would not keep anything you care about in it. Seems analog.

Context engineering is the work of building everything the model sits inside. Most of the writing on the term is about retrieval pipelines and agent memory and how to pack a context window, which is fine but not what I want to talk about. I mean the unglamorous human version: the documents, schemas, logs, and guardrails you maintain by hand so that a stateless model can do useful work on a real codebase over weeks. The prompt is what you say. The context is everything that is true when you say it. If you only invest in the first, you get good single answers and an incoherent project. If you invest in the second, the individual prompts barely matter, because the model is working inside something that already makes sense.

The first part of that is deciding the shape of the thing before you let the model generate any of it. I write the schema and the API rules first, before any feature work. Not because the model needs them, though it does, but because writing them is how I find out whether I actually understand what I am building. A vague idea survives in your head indefinitely. It does not survive being written down as a concrete data model with concrete endpoints. If I cannot specify the contract, I do not understand the feature yet, and no amount of clever prompting will rescue a thing I have not understood. Once the contract exists, the model can generate against it, and so can the next session, and the two will agree because they are both working from the same fixed reference rather than from two slightly different guesses.

The harder part is that the model has no memory, and you have to supply it yourself. Every new session starts blind. It does not know what you decided yesterday or why. So the context you carry between sessions has to live in files, not in your head and not in a chat history that gets truncated. Two documents do most of that work for me. The first is a handoff document that says where the project currently stands, what is done, what is in progress, and what the next session should pick up, so a fresh session can read it and continue instead of relearning the whole codebase from scratch. The second is an append-only decision log. Every time I take a shortcut because of some constraint at the time, or pick one approach over another for a reason that is not obvious from the code, it goes in the log and never gets edited out. The value is in the append-only part. Six weeks later, when something looks wrong, the log tells me whether it was a mistake or a deliberate trade I made for a reason I have since forgotten. Without it, every old decision becomes a small mystery, and you lose time re-arguing things you already settled.

What this setup actually buys you is fewer hallucinations, and it is worth being precise about why. The model does not invent things because you phrased the prompt badly. It invents things to fill gaps it cannot see across. When the schema is undefined and the past decisions are undocumented, those gaps are everywhere, and the model papers over them with plausible guesses. When the contract is fixed and the decisions are written down, there is far less room to guess, because the answer is already on the page. Good context makes the model more accurate by removing the empty space the inaccuracy comes from.

The last piece is making sure unfinished or risky work cannot slip by unnoticed, which matters more as the model writes more of the code. Anything on the frontend that is faked, stubbed, or temporary gets a hardcoded warning sitting right there in the interface, and the warning only comes out when the underlying thing is actually finished. That way a placeholder cannot quietly graduate into something that looks done. New features have to pass backtests against the existing ones, so that adding something cannot silently break what already worked. And there is a hard line that some parts are not the model’s to finish. Anything touching security, or anything with legal exposure, gets reviewed and tested by a human every time, regardless of how confident the generated version looks.

That last rule is the one I would not bend. The useful division of labor is not a percentage of how much code the model writes against how much I write, though the model does write most of it. It is a division by risk. The model handles the high-volume, well-specified work where a mistake is cheap and visible. Humans keep the parts where a mistake is expensive or hard to undo. The human stays in the loop. The aim is to spend their attention on the places that need it, rather than spreading it thin over code the model could have written correctly on its own.

None of this is worth doing for a script you will run once and delete. The overhead only pays off on something you have to keep, extend, and trust later. But for anything in that category, the wording of any single prompt turns out to be one of the least important decisions you make. What matters is whether the model is working inside a structure that holds its shape after you close the laptop. Prompt engineering gets you a good answer now. Context engineering is what makes the answer still fit a month later.

The usual way around this is a method from Longstaff and Schwartz (2001). You simulate a large number of possible price paths for the underlying stock, then walk backward from expiry. At each step you look at the paths where exercising is currently worth something, and ask whether it is better to exercise now or hold on. You estimate the value of holding by fitting a small regression against the current price, using the cash flows further along each path. Do that at every step and you get an exercise rule and a price out the other side.

There is a quiet problem in the simple version of this. The same simulated paths get used twice, once to fit the exercise rule and again to compute the final price. When you fit a rule and then score it on the exact data you fit it on, the score comes out flattering. The regression finds coefficients that happen to work well on those particular paths, and then you reward it for working well on those paths. The price comes out biased upward, and you have no easy way to see by how much.

Most of the time the effect is small and you might never notice it. I noticed because of one contract that broke. An out-of-the-money put on TCS came out priced about 4.78 below its European equivalent. That is not a rounding error, it is impossible. An American option can never be worth less than the matching European one, because if you hold an American option and simply choose never to exercise early, you get the European payoff anyway. A negative early-exercise premium means the math has gone wrong somewhere, and in this case the somewhere was the rule scoring itself on its own paths.

The fix is the sort of thing you would do to test any model. Build the exercise rule on one set of simulated paths, then throw those paths away and price the option on a completely fresh set the rule has never seen. Now the rule has to perform on data it was not fitted to, which is the only honest check of whether it is any good. This is the out-of-sample scheme from Rasmussen (2005). With it, the TCS put went from 4.78 below its European value to about 0.14 below, which is close enough to zero to be simulation noise. The impossible answer went away.

The fix is not completely free. For some well-behaved contracts the simple version has two opposite errors that partly cancel each other, and removing one of them slightly widens the average error. I think that is a good trade. I would rather have a number I can trust to be a sensible lower bound than a number that is closer on average but occasionally returns something that cannot be true.

The other half of being honest about this pricer is the data. I wanted to price contracts on real Indian stocks, but NSE blocks automated access to its live option chain. You can send the request, but it comes back with an empty record, and that holds whether you use the common Python libraries or send the request yourself with ordinary browser headers. Rather than build a heavy headless-browser scraper to get around that, I used what I could get cleanly. The spot prices are real, pulled from Yahoo Finance. The volatility is real, computed from each stock’s own recent daily returns. The expiry dates are real, following NSE’s last-Thursday convention. The one piece I generate myself is the ladder of strike prices, built from NSE’s published rules for how far apart strikes sit at a given price level. So the pricing runs on real inputs, against a strike grid that matches what NSE would actually list, even though I had to construct that grid rather than read it off the exchange.

The pricer validates against the original Longstaff-Schwartz paper and against a separate finite-difference calculation, and the numbers agree. But the part I care about more is that when it is wrong, it tends to be wrong in a direction I understand, rather than confidently handing back something that cannot happen.

The first kernel does closed-form Black-Scholes pricing. One thread per contract, no communication between threads, nothing depending on anything else. It prices a million contracts in about half a millisecond, roughly 270 times faster than the single-threaded CPU version I wrote as a baseline. That number looks good until you think about what the kernel is doing. Each thread reads a few numbers, runs them through a formula, and writes the result. There is almost no arithmetic involved relative to the amount of data being moved around. So most of that half millisecond goes to reading inputs and writing outputs, and the compute units sit mostly idle. The 270x is real, but it comes from the memory system, not from the math. The card was barely trying.

The second kernel is the one I actually cared about. It prices American options, which are harder than European ones because the holder can exercise early, on any day before expiry. To price that you have to work out, at every point in the option’s life, whether exercising now is worth more than holding on. The standard method (Longstaff and Schwartz, 2001) runs a Monte Carlo simulation of price paths and then walks backward from expiry, fitting a small regression at each step to estimate the value of holding. I run one thread per simulated path, so the simulation itself is very parallel, hundreds of thousands of paths moving at once.

The result was a 7x speedup over the CPU. After 270x on the easy kernel, 7x felt like a letdown, and for a while I assumed I had written something inefficient.

The problem is not the GPU. It is the shape of the algorithm. Walking backward through the option’s life is a sequence of steps, and each step depends on the one after it, so they cannot run at the same time. On top of that, every step needs to solve a tiny 3x3 system of equations to fit the regression. A 3x3 solve is too small to be worth doing on the GPU, so I hand it back to the CPU each time. That means the program runs a burst of work on the GPU, stops, waits for the CPU to do a little arithmetic, hands control back, and repeats, fifty times per contract. While the GPU kernel is actually running it sits in the high 80s for utilization, which is fine. The time is lost in the gaps, in all the stopping and starting and waiting.

So I ended up with two kernels that fail to push the card for opposite reasons. The simple one finishes too fast to stress anything except memory. The complicated one keeps interrupting itself to check in with the CPU. In neither case is the limit the card’s ability to compute, which is the thing I set out to test. The 4050 has plenty of headroom in both. It is just that one workload never asks for it and the other one keeps pausing before it can.

This is the part I did not expect when I started. I had a vague picture of a GPU as a thing you feed work to, and if you feed it enough, the work goes fast. In practice the card is rarely the bottleneck. You hit the speed of memory, or you hit parts of the problem that have to happen in order, long before you hit the limit of the arithmetic. Picking a problem that genuinely saturates the compute turns out to be its own skill, and a closed-form pricer or a sequential Monte Carlo is not it.

The fix for the slow kernel is reasonably clear. The reason I bounce out to the CPU is that the 3x3 solve is awkward to do on the GPU, but awkward is not the same as impossible. If I write the solve directly into the kernel and keep it on the device, the whole backward pass stays on the card and the stop-start pattern goes away. I expect that to recover most of the lost time. That is the next thing I want to try.

None of this made the pricer wrong. It validates against the original Longstaff-Schwartz paper and against a finite-difference reference, and the numbers line up. It just means the project taught me something other than what I went in for. I wanted to find the ceiling of the card. What I found instead was that for two fairly ordinary workloads, the card was never the thing in my way.

The problem showed up before I had written any analysis. GH Archive stores its data as hourly files. Each file is a gzipped block of JSON containing every public event that happened on GitHub in that hour, across every repository, all mixed together. You download by time, an hour or a range of hours, and that is the only axis you get. There is no way to ask the server for just one repository. If you want Apache Spark, you download the hour and Spark’s events are somewhere inside it, next to a few hundred thousand events from everyone else.

For a six month window that adds up fast. Recent hourly files run over a hundred megabytes compressed each, and there are twenty four of them a day. The stretch I wanted came to something like half a terabyte. My laptop had about a hundred gigabytes free. So the data was several times too big to hold, and almost all of it was data I did not want, since I only cared about one repository out of millions.

The obvious approach is to download everything, then filter, then delete the rest. That does not work when the download alone is several times your free space. You would fill the disk long before you reached the filtering step. I needed to filter while downloading, not after, so that the stuff I was throwing away never had to land on disk in the first place.

That is the whole idea behind the tool I ended up writing, which I called gharc. It works one hour at a time. It downloads a single hourly file into a small temporary buffer, reads through it, keeps only the events matching the repositories and event types I asked for, writes those out, and deletes the raw file before moving to the next hour. The raw data passes through memory and leaves. At no point does the full dataset exist on the disk. Disk usage stays flat at well under a hundred megabytes no matter how many months I process, because the only thing that grows is the filtered output, which for a single repository is small.

A few details mattered more than I expected once I started running it for real.

The first was the filtering itself. Fully parsing every JSON record just to check which repository it belongs to is slow, and most records are going to be thrown away anyway. So before doing a real parse, the tool does a cheap substring check for the repository name on the raw line. If the name is not even present as text, there is no point parsing the record, and it gets skipped. Only the lines that survive that check get parsed properly. Since the keep rate for one repository is very low, this skips the large majority of the work.

The second was that residential internet is not reliable over long runs. Downloading hundreds of files in a row, something will eventually drop. Rather than restart a file from the beginning when that happens, the tool uses HTTP range requests to resume from where it stopped. Over a multi hour run that was the difference between finishing and not.

The third was the output format. I write the filtered events to Parquet rather than JSON. Parquet stores data by column instead of by row and compresses well, so the output files end up far smaller than the equivalent JSON, and they load straight into Pandas, Polars, or Spark with no conversion step. Given that the point of the exercise was to then analyze the data in Spark, that lined up nicely.

What I find a little funny in hindsight is that the course was about big data analysis, and the analysis was never really the hard part. The hard part was getting the data into a shape and size that a normal laptop could hold at all. The actual Spark analysis at the end ran on a few hundred megabytes of filtered Parquet, which is nothing. Almost all of the work went into the step before that, into not drowning in the half terabyte I had to pass through to get there.

The tool turned out general enough that it is not really tied to Spark or to my assignment. You give it a date range, a list of repositories, optionally a list of event types, and it hands you a compact dataset for exactly those, pulled out of the firehose without needing the room to store the firehose. That constraint, not having the disk space, is what made it worth building.