This started as a mini project for a big data course in my seventh semester. The brief was open-ended, which is the sort of freedom a student should probably be protected from. I decided to analyze the Apache Spark codebase using Apache Spark, on the grounds that it sounded clever and I had not yet thought about the logistics. To do that I needed the public history of activity on the Spark repository, and the obvious place to get it is GH Archive, a project that records the public GitHub event timeline and makes it available to download.

The problem showed up before I had written any analysis. GH Archive stores its data as hourly files. Each file is a gzipped block of JSON containing every public event that happened on GitHub in that hour, across every repository, all mixed together. You download by time, an hour or a range of hours, and that is the only axis you get. There is no way to ask the server for just one repository. If you want Apache Spark, you download the hour and Spark’s events are somewhere inside it, next to a few hundred thousand events from everyone else.

For a six month window that adds up fast. Recent hourly files run over a hundred megabytes compressed each, and there are twenty four of them a day. The stretch I wanted came to something like half a terabyte. My laptop had about a hundred gigabytes free. So the data was several times too big to hold, and almost all of it was data I did not want, since I only cared about one repository out of millions.

The obvious approach is to download everything, then filter, then delete the rest. That does not work when the download alone is several times your free space. You would fill the disk long before you reached the filtering step. I needed to filter while downloading, not after, so that the stuff I was throwing away never had to land on disk in the first place.

That is the whole idea behind the tool I ended up writing, which I called gharc. It works one hour at a time. It downloads a single hourly file into a small temporary buffer, reads through it, keeps only the events matching the repositories and event types I asked for, writes those out, and deletes the raw file before moving to the next hour. The raw data passes through memory and leaves. At no point does the full dataset exist on the disk. Disk usage stays flat at well under a hundred megabytes no matter how many months I process, because the only thing that grows is the filtered output, which for a single repository is small.

A few details mattered more than I expected once I started running it for real.

The first was the filtering itself. Fully parsing every JSON record just to check which repository it belongs to is slow, and most records are going to be thrown away anyway. So before doing a real parse, the tool does a cheap substring check for the repository name on the raw line. If the name is not even present as text, there is no point parsing the record, and it gets skipped. Only the lines that survive that check get parsed properly. Since the keep rate for one repository is very low, this skips the large majority of the work.

The second was that residential internet is not reliable over long runs. Downloading hundreds of files in a row, something will eventually drop. Rather than restart a file from the beginning when that happens, the tool uses HTTP range requests to resume from where it stopped. Over a multi hour run that was the difference between finishing and not.

The third was the output format. I write the filtered events to Parquet rather than JSON. Parquet stores data by column instead of by row and compresses well, so the output files end up far smaller than the equivalent JSON, and they load straight into Pandas, Polars, or Spark with no conversion step. Given that the point of the exercise was to then analyze the data in Spark, that lined up nicely.

What I find a little funny in hindsight is that the course was about big data analysis, and the analysis was never really the hard part. The hard part was getting the data into a shape and size that a normal laptop could hold at all. The actual Spark analysis at the end ran on a few hundred megabytes of filtered Parquet, which is nothing. Almost all of the work went into the step before that, into not drowning in the half terabyte I had to pass through to get there.

The tool turned out general enough that it is not really tied to Spark or to my assignment. You give it a date range, a list of repositories, optionally a list of event types, and it hands you a compact dataset for exactly those, pulled out of the firehose without needing the room to store the firehose. That constraint, not having the disk space, is what made it worth building.