PageRenderTime 56ms CodeModel.GetById 25ms RepoModel.GetById 0ms app.codeStats 0ms

/blog/2021/05/12/why-you-shouldn't-use-git-lfs/index.html

https://github.com/indygreg/indygreg.github.com
HTML | 262 lines | 221 code | 30 blank | 11 comment | 0 complexity | 1cd851d42fa513839b57d926ce1e3965 MD5 | raw file
    

  1. 

  2. 

  3. 

  4. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" 
    5. 

  5.     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
    6. 

  6. <!--
    7. 

  7. Design by Free CSS Templates
    8. 

  8. http://www.freecsstemplates.org
    9. 

  9. Released for free under a Creative Commons Attribution 2.5 License
    10. 

  10. 

  11. Name       : Pollinating  
    12. 

  12. Description: A two-column, fixed-width design with dark color scheme.
    13. 

  13. Version    : 1.0
    14. 

  14. Released   : 20101114
    15. 

  15. 

  16. -->
    17. 

  17. <html xmlns="http://www.w3.org/1999/xhtml">
    18. 

  18.   <head>
    19. 

  19.     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    20. 

  20.     
    21. 

  21.   <title>Gregory Szorc's Digital Home
    22. 

  22.   | Why You Shouldn't Use Git LFS
    23. 

  23. </title>
    24. 

  24. <link rel="alternate" type="application/rss+xml" title="RSS 2.0" href="/blog/feed" />
    25. 

  25. <link rel="alternate" type="application/atom+xml" title="Atom 1.0"
    26. 

  26. href="/blog/feed/atom" />
    27. 

  27. <link rel="stylesheet" href="/style/style.css" type="text/css" />
    28. 

  28. <link rel="stylesheet" href="/css/pygments_murphy.css" type="text/css" />
    29. 

  29. 

  30. 

  31.   </head>
    32. 

  32.   <body>
    33. 

  33.     <div id="wrapper">
    34. 

  34.       
    35. 

  35.   <div id="menu">
    36. 

  36.   <ul>
    37. 

  37.     <li><a href="/">Home</a></li>
    38. 

  38.     <li><a href="/blog/">Blog</a></li>
    39. 

  39.     <li><a href="/notes">Notes</a></li>
    40. 

  40.     <li><a href="/work.html">Work</a></li>
    41. 

  41.     <li><a href="/skills.html">Skills</a></li>
    42. 

  42.     <li><a href="/thoughts.html">Thoughts</a></li>
    43. 

  43.     <li><a href="/resume.pdf">Resume</a></li>
    44. 

  44.   </ul>
    45. 

  45. </div>
    46. 

  46. 

  47. 

  48.       <div id="page">
    49. 

  49.         <div id="page-bgtop">
    50. 

  50.           <div id="page-bgbtm">
    51. 

  51.               <div id="content">
    52. 

  52.                 
    53. 

  53. 

  54. <div class="blog_post">
    55. 

  55.   <a name="why-you-shouldn't-use-git-lfs"></a>
    56. 

  56.   <h2 class="blog_post_title"><a href="/blog/2021/05/12/why-you-shouldn't-use-git-lfs" rel="bookmark" title="Permanent Link to Why You Shouldn't Use Git LFS">Why You Shouldn't Use Git LFS</a></h2>
    57. 

  57.   <small>May 12, 2021 at 10:30 AM | categories: 
    58. 

  58. 

  59. <a href='/blog/category/mercurial'>Mercurial</a>, <a href='/blog/category/git'>Git</a>
    60. 

  60. </small><p/>
    61. 

  61.   <div class="post_prose">
    62. 

  62.     
    63. 

  63.   <p>I have long held the opinion that you should avoid Git LFS if possible.
    64. 

  64. Since people keeping asking me why, I figured I'd capture my thoughts
    65. 

  65. in a blog post so I have something to refer them to.</p>
    66. 

  66. <p>Here are my reasons for not using Git LFS.</p>
    67. 

  67. <h2>Git LFS is a Stop Gap Solution</h2>
    68. 

  68. <p>Git LFS was developed outside the official Git project to fulfill a
    69. 

  69. very real market need that Git didn't/doesn't handle large files very
    70. 

  70. well.</p>
    71. 

  71. <p>I believe it is inevitable that Git will gain better support for
    72. 

  72. handling of large files, as this seems like a critical feature for
    73. 

  73. a popular version control tool.</p>
    74. 

  74. <p>If you make this long bet, LFS is only an interim solution and its
    75. 

  75. value proposition disappears after Git has better native support
    76. 

  76. for large files.</p>
    77. 

  77. <p>LFS as a stop gap solution would be tolerable except for the fact
    78. 

  78. that...</p>
    79. 

  79. <h2>Git LFS is a One Way Door</h2>
    80. 

  80. <p>The adoption or removal of Git LFS in a repository is an irreversible
    81. 

  81. decision that requires rewriting history and losing your original 
    82. 

  82. commit SHAs.</p>
    83. 

  83. <p>In some contexts, rewriting history is tolerable. In many others, it
    84. 

  84. is an extremely expensive proposition. My experience maintaining
    85. 

  85. version control in professional contexts aligns with the opinion
    86. 

  86. that rewriting history is expensive and should only be considered a
    87. 

  87. measure of last resort. Maybe if tools made it easier to rewrite
    88. 

  88. history without the negative consequences (e.g. GitHub would redirect
    89. 

  89. references to old SHA1 in URLs and API calls) I would change my opinion
    90. 

  90. here. Until that day, the drawbacks of losing history are just too
    91. 

  91. high to stomach for many.</p>
    92. 

  92. <p>The reason adoption or removal of LFS is irreversible is due to the
    93. 

  93. way Git LFS works. What LFS does is change the blob content that a
    94. 

  94. Git commit/tree references. Instead of the content itself, it stores
    95. 

  95. a pointer to the content. At checkout and commit time, LFS blobs/records
    96. 

  96. are treated specially via a mechanism in Git that allows content to be
    97. 

  97. rewritten as it moves between Git's core storage and its materialized
    98. 

  98. representation. (The same filtering mechanism is responsible for
    99. 

  99. normalizing line endings in text files. Although that feature is built
    100. 

  100. into the core Git product and doesn't work exactly the same way. But
    101. 

  101. the principles are the same.)</p>
    102. 

  102. <p>Since the LFS pointer is part of the Merkle tree that a Git commit
    103. 

  103. derives from, you can't add or remove LFS from a repo without rewriting
    104. 

  104. existing Git commit SHAs.</p>
    105. 

  105. <p>I want to explicitly call out that even if a rewrite is acceptable in
    106. 

  106. the short term, things may change in the future. If you adopt LFS
    107. 

  107. today, you are committing to a) running an LFS server forever b)
    108. 

  108. incurring a history rewrite in the future in order to remove LFS from
    109. 

  109. your repo, or c) ceasing to provide an LFS server and locking out people
    110. 

  110. from using older Git commits. I don't think any of these are great
    111. 

  111. options: I would prefer if there were a way to offboard from LFS in
    112. 

  112. the future with minimal disruption. This is theoretically possible, but
    113. 

  113. it requires the Git core product to recognize LFS blobs/records natively.
    114. 

  114. There's no guarantee this will happen. So adoption of Git LFS is a one
    115. 

  115. way door that can't be easily reversed.</p>
    116. 

  116. <h2>LFS is More Complexity</h2>
    117. 

  117. <p>LFS is more complex for Git end users.</p>
    118. 

  118. <p>Git users have to install, configure, and sometimes know about the
    119. 

  119. existence of Git LFS. Version control should <em>just work</em>. Large file
    120. 

  120. handling should <em>just work</em>. End-users shouldn't have to care that
    121. 

  121. large files are handled slightly differently from small files.</p>
    122. 

  122. <p>The usability of Git LFS is generally pretty good. However, there's an
    123. 

  123. upper limit on that usability as long as LFS exists outside the core
    124. 

  124. Git product. And LFS will likely never be integrated into the core Git
    125. 

  125. product because the Git maintainers know that LFS is only a stop gap
    126. 

  126. solution. They would rather solve large files storage <em>correctly</em> than
    127. 

  127. ~forever carry the legacy baggage of having to support LFS in the core
    128. 

  128. product.</p>
    129. 

  129. <p>LFS is more complexity for Git server operators as well. Instead of
    130. 

  130. a self-contained Git repository and server to support, you now have
    131. 

  131. to support a likely separate HTTP server to facilitate LFS access.
    132. 

  132. This isn't the hardest thing in the world, especially since we're
    133. 

  133. talking about key-value blob storage, which is arguably a solved problem.
    134. 

  134. But it's another piece of infrastructure to support and secure and it
    135. 

  135. increases the surface area of complexity instead of minimizing it.
    136. 

  136. As a server operator, I would much prefer if the large file storage
    137. 

  137. were integrated into the core Git product and I simply needed to provide
    138. 

  138. some settings for it to <em>just work</em>.</p>
    139. 

  139. <h2>Mercurial Does LFS Slightly Better</h2>
    140. 

  140. <p>Since I'm a maintainer of the Mercurial version control tool, I thought
    141. 

  141. I'd throw out how Mercurial handles large file storage better than
    142. 

  142. Git. Mercurial's large file handling isn't great, but I believe it
    143. 

  143. is strictly better with regards to the trade-offs of adopting large
    144. 

  144. file storage.</p>
    145. 

  145. <p>In Mercurial, use of LFS is a dynamic feature that server/repo operators
    146. 

  146. can choose to enable or disable whenever they want. When the Mercurial
    147. 

  147. server sends file content to a client, presence of external/LFS storage
    148. 

  148. is a <em>flag</em> set on that file revision. Essentially, the flag says <em>the
    149. 

  149. data you are receiving is an LFS record, not the file content itself</em> and
    150. 

  150. the client knows how to resolve that record into content.</p>
    151. 

  151. <p>Conceptually, this is little different from Git LFS records in terms of
    152. 

  152. content resolution. However, the big difference is this flag is part
    153. 

  153. of the repository interchange data, not the core repository data as it
    154. 

  154. is with Git. Since this flag isn't part of the Merkle tree used to
    155. 

  155. derive the commit SHA, adding, removing, or altering the content of the
    156. 

  156. LFS records doesn't require rewriting commit SHAs. The tracked content
    157. 

  157. SHA - the data now stored in LFS - is still tracked as part of the Merkle
    158. 

  158. tree, so the integrity of the commit / repository can still be verified.</p>
    159. 

  159. <p>In Mercurial, the choice of whether to use LFS and what to use LFS for
    160. 

  160. is made by the server operator and settings can change over time. For
    161. 

  161. example, you could start with no use of LFS and then one day decide to
    162. 

  162. use LFS for all file revisions larger than 10 MB. Then a year later
    163. 

  163. you lower that to all revisions larger than 1 MB. Then a year after
    164. 

  164. that Mercurial gains better <em>native</em> support for large files and
    165. 

  165. you decide to stop using LFS altogether.</p>
    166. 

  166. <p>Also in Mercurial, it is possible for clients to push a large file
    167. 

  167. <em>inline</em> as part of the push operation. When the server sees that
    168. 

  168. large file, it can be like <em>this is a large file: I'm going to add
    169. 

  169. it to the blob store and advertise it as LFS</em>. Because the large
    170. 

  170. file record isn't part of the Merkle tree, you can have nice things
    171. 

  171. like this.</p>
    172. 

  172. <p>I suspect it is only a matter of time before Git's wire protocol learns
    173. 

  173. the ability to dynamically advertise <em>remote servers</em> for content
    174. 

  174. retrieval and this feature will be leveraged for better large file
    175. 

  175. handling. Until that day, I suppose we're stuck with having to
    176. 

  176. rewrite history with LFS and/or funnel large blobs through Git natively,
    177. 

  177. with all the pain that entails.</p>
    178. 

  178. <h2>Conclusion</h2>
    179. 

  179. <p>This post summarized reasons to avoid Git LFS. Are there justifiable
    180. 

  180. scenarios for using LFS? Absolutely! If you insist on using Git and
    181. 

  181. insist on tracking many <em>large</em> files in version control, you
    182. 

  182. should definitely consider LFS. (Although, if you are a heavy user
    183. 

  183. of large files in version control, I would consider Plastic SCM instead,
    184. 

  184. as they seem to have the most mature solution for large files handling.)</p>
    185. 

  185. <p>The main point of this post is to highlight some drawbacks with
    186. 

  186. using Git LFS because Git LFS is most definitely not a magic bullet. If
    187. 

  187. you can stomach the short and long term effects of Git LFS adoption, by
    188. 

  188. all means, use Git LFS. But please make an informed decision either way.</p>
    189. 

  189. 

  190.   </div>
    191. 

  191. </div>
    192. 

  192. 

  193. 

  194. 

  195. 

  196.               </div>
    197. 

  197.               
    198. 

  198.           <div id="sidebar">
    199. 

  199.           <ul>
    200. 

  200.             <li>
    201. 

  201.               <h2>Categories</h2>
    202. 

  202.               <ul>
    203. 

  203.                 <li><a href="/blog/category/apple">Apple</a></li>
    204. 

  204.                 <li><a href="/blog/category/bugzilla">Bugzilla</a></li>
    205. 

  205.                 <li><a href="/blog/category/ci">CI</a></li>
    206. 

  206.                 <li><a href="/blog/category/clang">Clang</a></li>
    207. 

  207.                 <li><a href="/blog/category/docker">Docker</a></li>
    208. 

  208.                 <li><a href="/blog/category/firefox">Firefox</a></li>
    209. 

  209.                 <li><a href="/blog/category/git">Git</a></li>
    210. 

  210.                 <li><a href="/blog/category/javascript">JavaScript</a></li>
    211. 

  211.                 <li><a href="/blog/category/mercurial">Mercurial</a></li>
    212. 

  212.                 <li><a href="/blog/category/mozreview">MozReview</a></li>
    213. 

  213.                 <li><a href="/blog/category/mozilla">Mozilla</a></li>
    214. 

  214.                 <li><a href="/blog/category/personal">Personal</a></li>
    215. 

  215.                 <li><a href="/blog/category/programming">Programming</a></li>
    216. 

  216.                 <li><a href="/blog/category/puppet">Puppet</a></li>
    217. 

  217.                 <li><a href="/blog/category/pyoxidizer">PyOxidizer</a></li>
    218. 

  218.                 <li><a href="/blog/category/python">Python</a></li>
    219. 

  219.                 <li><a href="/blog/category/review-board">Review Board</a></li>
    220. 

  220.                 <li><a href="/blog/category/rust">Rust</a></li>
    221. 

  221.                 <li><a href="/blog/category/sync">Sync</a></li>
    222. 

  222.                 <li><a href="/blog/category/browsers">browsers</a></li>
    223. 

  223.                 <li><a href="/blog/category/build-system">build system</a></li>
    224. 

  224.                 <li><a href="/blog/category/code-review">code review</a></li>
    225. 

  225.                 <li><a href="/blog/category/compilers">compilers</a></li>
    226. 

  226.                 <li><a href="/blog/category/internet">internet</a></li>
    227. 

  227.                 <li><a href="/blog/category/logging">logging</a></li>
    228. 

  228.                 <li><a href="/blog/category/mach">mach</a></li>
    229. 

  229.                 <li><a href="/blog/category/make">make</a></li>
    230. 

  230.                 <li><a href="/blog/category/misc">misc</a></li>
    231. 

  231.                 <li><a href="/blog/category/movies">movies</a></li>
    232. 

  232.                 <li><a href="/blog/category/packaging">packaging</a></li>
    233. 

  233.                 <li><a href="/blog/category/pymake">pymake</a></li>
    234. 

  234.                 <li><a href="/blog/category/security">security</a></li>
    235. 

  235.                 <li><a href="/blog/category/sysadmin">sysadmin</a></li>
    236. 

  236.                 <li><a href="/blog/category/testing">testing</a></li>
    237. 

  237.               </ul>
    238. 

  238.             </li>
    239. 

  239.           </ul>
    240. 

  240.         </div>
    241. 

  241. 

  242. 

  243. 

  244.               <div style="clear: both;">&nbsp;</div>
    245. 

  245.           </div>
    246. 

  246.         </div>
    247. 

  247.       </div>
    248. 

  248.       <div id="footer">
    249. 

  249.         
    250. 

  250.   <hr/>
    251. 

  251.   <p>Copyright (c) 2012- Gregory Szorc. All rights reserved. Design by <a href="http://www.freecsstemplates.org/"> CSS Templates</a>.</p>
    252. 

  252. 

  253. 

  254.       </div>
    255. 

  255.     </div>
    256. 

  256.   </body>
    257. 

  257. </html>
    258. 

  258. 

  259. 

  260. 

  261. 

  262. 

  263.