Git, the Linux kernel, and other projects. 2


So every now and then I hear:

– We’re falling short on git. It wasn’t designed for big projects. [often in the context of game development]
– Is your project as big as the Linux Kernel? No? I thought so.

I find this extremely misleading. Not because the Linux Kernel isn’t big, but because there are different kinds of big. Git was specifically designed by Linus Trovalds to handle the Linux kernel’s source code.

The Linux kernel src has the following properties:

  1. It’s mostly composed of text. i.e. no binary data like pictures, sound files, 3D models, or other formats.
  2. Low size of HEAD: The entire checkout of a repository is less than 200MB. I’m not talking about the size of the “.git” folder, but rather of the files once you check out the HEAD.
  3. Small individual file sizes.
  4. Relatively flat: It’s a few folders deep, then lots of files under that folder.
  5. To be used mostly on Linux.

This is pretty much the opposite of a game development repo:

  1. Lots of binary files whose versioning needs to be in sync with the source code.
  2. Gigantic size of HEAD: The entire check out of the repository can be >1GB or > 2GB easily for indies alone. >50GB for the bare minimum an AAA game needs to keep in sync.
  3. Big individual file sizes.
  4. Deeply nested folder structures, with lots of files and lots of files within them.
  5. Cross platform development (or when it’s not, it’s predominantly Windows).

The last point is something that really irritates me. In Windows, large repos are unbearable slow, pushing produces inexplicable EOL errors that need to disable compression, committing a lot of files together reaches Windows cmd line limits and fails, committing a file with a long filepath fails, git updates really lack behind because they need to be ported; among the many issues I periodically stumble.

Hopefully these issues should go away with the Windows 10’s Ubuntu bash but I haven’t tried it yet.

But the problem is that Linus Trovalds doesn’t care about these issues, because it runs fine on Linux and works well for his kernel. Perhaps he is in his own right to think like that (after all, I don’t think he ever advertised the tool to be adopted widespread?), but I’m not ok with git fans mocking other projects because “they are not as big as the Linux kernel” or that “we should move to Linux”.

My thought to those replies is always: “Perhaps I should move out of git instead”. I want Linux: The Year of the Desktop to succeed, I really do, particularly since Windows 10 started to take over my computer. But even today the best graphics tools are mostly running on Windows. X11 is no closer to dying it was a year ago and I still can’t watch Tear-free videos. Now that Mesa’s latest commit is finally competent enough for me to use it, I’ve compiled the debug version to debug whatever issue I encounter (I see some really hard to catch timing errors; I will catch it eventually).

Things like LFS Git help with large files, but it doesn’t solve all the problems. To be honest the biggest attraction of git is GitHub. Great 3rd party extensions (like ZenHub), great 3rd party CI services (both free and paid), an excellent issue tracker and superb wiki. The competition is miles behind them. If I chose git it would be because of GitHub and its ecosystem, not because of git.

So, point of this article: if git is right for you, great. But don’t pretend it’s the right fit for everybody.


Leave a comment

Your email address will not be published. Required fields are marked *

2 thoughts on “Git, the Linux kernel, and other projects.

  • CryHam

    Any thoughts on how Hg compares to Git on Windows in those regards?
    Have you been using TortoiseHg or something else for Ogre?
    Our repo for StuntRally was about 2GB finally (actually two: one 500MB source+data and second just for tracks 1,5GB). Luckily we didn’t have any of the above problems with it using Git.

  • Matias Post author

    In my experience, on Windows Hg destroys git.
    For starters Hg never has that issue that the repo may be in an inconsistent state; because all of their operations are atomic, while git may require multiple commands to perform some advanced operations.

    I haven’t run into any performance, compatibility or stability issues; unlike git which I run into these issues on a weekly basis.

    Furthermore TortoiseHg is a wonderful and fast GUI to use that is available on Windows, Linux & macOS, giving a consistent experience.

    Furthermore in terms of large data files, Hg has more mature “big files” extensions whereas git’s are much more recent.

    Now the comparison on Linux & macOS is a different deal. Git is in a much more even ground there performance and stability wise; and macOS has SourceTree (which unlike it’s Windows counterpart, it’s still a nice and fast GUI) and in Linux so far the best free GUI has been Git Cola (very rough to get used to, I rejected it at first; but it gets the job done well once you get how it works).
    I still prefer Hg for personal taste, but I admit that if the project is mostly Linux/macOS, then the benefits of having GitHub may outweight the benefits of Hg.