Is there really no use for MD5 anymore? Announcing the arrival of Valued Associate #679: Cesar Manara Unicorn Meta Zoo #1: Why another podcast?How many trials does it take to break HMAC-MD5?What differentiates a password hash from a cryptographic hash besides speed?A question regarding relevance of vulnerability of MD5 when linking multiple records togetherCould a very long password theoretically eliminate the need for a slow hash?TCR hash functions from MD5Collision attacks on digital signaturesChecksum vs. non-cryptographic hashIs using a broken SHA-1 for password hashing secure?Unable to implement Client and Server Side Hashing (Validation problem)Keyspace in truncated MD5 hash?Very difficult hashing function?

finding a tangent line to a parabola

A Paper Record is What I Hamper

Will I lose my paid in full property

Why does Arg'[1. + I] return -0.5?

Older movie/show about humans on derelict alien warship which refuels by passing through a star

Is Diceware more secure than a long passphrase?

How to find the stem of any word?

How to find if a column is referenced in a computed column?

A strange hotel

Why doesn't the standard consider a template constructor as a copy constructor?

How can I wire a 9-position switch so that each position turns on one more LED than the one before?

Do I need to watch Ant-Man and the Wasp and Captain Marvel before watching Avengers: Endgame?

The weakest link

How much cash can I safely carry into the USA and avoid civil forfeiture?

How do I prove this combinatorial identity

Is it acceptable to use working hours to read general interest books?

Is there any pythonic way to find average of specific tuple elements in array?

Why do games have consumables?

Is this homebrew arcane communication device abusable?

What was Apollo 13's "Little Jolt" after MECO?

Are there moral objections to a life motivated purely by money? How to sway a person from this lifestyle?

How does the mezzoloth's teleportation work?

Intern got a job offer for same salary than a long term team member

Can I criticise the more senior developers around me for not writing clean code?



Is there really no use for MD5 anymore?



Announcing the arrival of Valued Associate #679: Cesar Manara
Unicorn Meta Zoo #1: Why another podcast?How many trials does it take to break HMAC-MD5?What differentiates a password hash from a cryptographic hash besides speed?A question regarding relevance of vulnerability of MD5 when linking multiple records togetherCould a very long password theoretically eliminate the need for a slow hash?TCR hash functions from MD5Collision attacks on digital signaturesChecksum vs. non-cryptographic hashIs using a broken SHA-1 for password hashing secure?Unable to implement Client and Server Side Hashing (Validation problem)Keyspace in truncated MD5 hash?Very difficult hashing function?










2












$begingroup$


I read an article about password schemes that makes two seemingly conflicting claims:




MD5 is broken; it’s too slow to use as a general purpose hash; etc



The problem is that MD5 is fast




I know that MD5 should not be used for password hashing, and that it also should not be used for integrity checking of documents. There are way too many sources citing MD5 preimaging attacks and MD5s low computation time.



However, I was under the impression that MD5 still can be used as a non-cryptgraphic hash function:



  1. Identifying malicious files, such as when Linux Mint's download servers were compromised and an ISO file was replaced by a malicious one; in this case you want to be sure that your file doesn't match; collision attacks aren't a vector here.

  2. Finding duplicate files. By MD5-summing all files in a directory structure it's easy to find identical hashes. The seemingly identical files can then be compared in full to check if they are really identical. Using SHA512 would make the process slower, and since we compare files in full anyway there is no risk in a potential false positive from MD5. (In a way, this would be creating a rainbow table where all the files are the dictionary)

There are checksums of course, but from my experience, the likelihood of finding two different files with the same MD5 hash is very low as long as we can rule out foul play.



When the password scheme article states that "MD5 is fast", it clearly refers to the problem that hashing MD5 is too cheap when it comes to hashing a large amount of passwords to find the reverse of a hash. But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?










share|improve this question









$endgroup$







  • 2




    $begingroup$
    The collision attack is the problem, however, MD5 still has pre-image resistance.
    $endgroup$
    – kelalaka
    7 hours ago















2












$begingroup$


I read an article about password schemes that makes two seemingly conflicting claims:




MD5 is broken; it’s too slow to use as a general purpose hash; etc



The problem is that MD5 is fast




I know that MD5 should not be used for password hashing, and that it also should not be used for integrity checking of documents. There are way too many sources citing MD5 preimaging attacks and MD5s low computation time.



However, I was under the impression that MD5 still can be used as a non-cryptgraphic hash function:



  1. Identifying malicious files, such as when Linux Mint's download servers were compromised and an ISO file was replaced by a malicious one; in this case you want to be sure that your file doesn't match; collision attacks aren't a vector here.

  2. Finding duplicate files. By MD5-summing all files in a directory structure it's easy to find identical hashes. The seemingly identical files can then be compared in full to check if they are really identical. Using SHA512 would make the process slower, and since we compare files in full anyway there is no risk in a potential false positive from MD5. (In a way, this would be creating a rainbow table where all the files are the dictionary)

There are checksums of course, but from my experience, the likelihood of finding two different files with the same MD5 hash is very low as long as we can rule out foul play.



When the password scheme article states that "MD5 is fast", it clearly refers to the problem that hashing MD5 is too cheap when it comes to hashing a large amount of passwords to find the reverse of a hash. But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?










share|improve this question









$endgroup$







  • 2




    $begingroup$
    The collision attack is the problem, however, MD5 still has pre-image resistance.
    $endgroup$
    – kelalaka
    7 hours ago













2












2








2





$begingroup$


I read an article about password schemes that makes two seemingly conflicting claims:




MD5 is broken; it’s too slow to use as a general purpose hash; etc



The problem is that MD5 is fast




I know that MD5 should not be used for password hashing, and that it also should not be used for integrity checking of documents. There are way too many sources citing MD5 preimaging attacks and MD5s low computation time.



However, I was under the impression that MD5 still can be used as a non-cryptgraphic hash function:



  1. Identifying malicious files, such as when Linux Mint's download servers were compromised and an ISO file was replaced by a malicious one; in this case you want to be sure that your file doesn't match; collision attacks aren't a vector here.

  2. Finding duplicate files. By MD5-summing all files in a directory structure it's easy to find identical hashes. The seemingly identical files can then be compared in full to check if they are really identical. Using SHA512 would make the process slower, and since we compare files in full anyway there is no risk in a potential false positive from MD5. (In a way, this would be creating a rainbow table where all the files are the dictionary)

There are checksums of course, but from my experience, the likelihood of finding two different files with the same MD5 hash is very low as long as we can rule out foul play.



When the password scheme article states that "MD5 is fast", it clearly refers to the problem that hashing MD5 is too cheap when it comes to hashing a large amount of passwords to find the reverse of a hash. But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?










share|improve this question









$endgroup$




I read an article about password schemes that makes two seemingly conflicting claims:




MD5 is broken; it’s too slow to use as a general purpose hash; etc



The problem is that MD5 is fast




I know that MD5 should not be used for password hashing, and that it also should not be used for integrity checking of documents. There are way too many sources citing MD5 preimaging attacks and MD5s low computation time.



However, I was under the impression that MD5 still can be used as a non-cryptgraphic hash function:



  1. Identifying malicious files, such as when Linux Mint's download servers were compromised and an ISO file was replaced by a malicious one; in this case you want to be sure that your file doesn't match; collision attacks aren't a vector here.

  2. Finding duplicate files. By MD5-summing all files in a directory structure it's easy to find identical hashes. The seemingly identical files can then be compared in full to check if they are really identical. Using SHA512 would make the process slower, and since we compare files in full anyway there is no risk in a potential false positive from MD5. (In a way, this would be creating a rainbow table where all the files are the dictionary)

There are checksums of course, but from my experience, the likelihood of finding two different files with the same MD5 hash is very low as long as we can rule out foul play.



When the password scheme article states that "MD5 is fast", it clearly refers to the problem that hashing MD5 is too cheap when it comes to hashing a large amount of passwords to find the reverse of a hash. But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?







hash md5






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked 7 hours ago









jornanejornane

1484




1484







  • 2




    $begingroup$
    The collision attack is the problem, however, MD5 still has pre-image resistance.
    $endgroup$
    – kelalaka
    7 hours ago












  • 2




    $begingroup$
    The collision attack is the problem, however, MD5 still has pre-image resistance.
    $endgroup$
    – kelalaka
    7 hours ago







2




2




$begingroup$
The collision attack is the problem, however, MD5 still has pre-image resistance.
$endgroup$
– kelalaka
7 hours ago




$begingroup$
The collision attack is the problem, however, MD5 still has pre-image resistance.
$endgroup$
– kelalaka
7 hours ago










5 Answers
5






active

oldest

votes


















5












$begingroup$


I know that MD5 should not be used for password hashing, and that it also should not be used for integrity checking of documents. There are way too many sources citing MD5 preimaging attacks and MD5s low computation time.




There is no published preimage attack on MD5 that is cheaper than a generic attack on any 128-bit hash function. But you shouldn't rely on that alone when making security decisions, because cryptography is tricky and adversaries are clever and resourceful and can find ways around it!




  1. Identifying malicious files, such as when Linux Mint's download servers were compromised and an ISO file was replaced by a malicious one; in this case you want to be sure that your file doesn't match; collision attacks aren't a vector here.



There are two issues here:




  1. If you got the MD5 hash from the same source as the ISO image, there's nothing that would prevent an adversary from replacing both the MD5 hash and the ISO image.



    To prevent this, you and the Linux Mint curators need two channels: one for the hashes which can't be compromised (but need only have very low bandwidth), and another for the ISO image (which needs high bandwidth) on which you can then use the MD5 hash in an attempt to detect compromise.



    There's another way to prevent this: Instead of using the uncompromised channel for the hash of every ISO image over and over again as time goes on—which means more and more opportunities for an attacker to subvert it—use it once initially for a public key, which is then used to sign the ISO images; then there's only one opportunity for an attacker to subvert the public key channel.




  2. Collision attacks may still be a vector in cases like this. Consider the following scenario:



    • I am an evil developer. I write two software packages, whose distributions collide under MD5. One of the packages is benign and will survive review and audit. The other one will surreptitiously replace your family photo album by erotic photographs of sushi.

    • The Linux Mint curators carefully scrutinize and audit everything they publish in their package repository and publish the MD5 hashes of what they have audited in a public place that I can't compromise.

    • The Linux Mint curators cavalierly administer the package distributions in their package repository, under the false impression that the published MD5 hashes will protect users.

    In this scenario, I can replace the benign package by the erotic sushi package, pass the MD5 verification with flying colors, and give you a nasty—and luscious—surprise when you try to look up photos of that old hiking trip you took your kids on.




  1. Finding duplicate files. By MD5-summing all files in a directory structure it's easy to find identical hashes. The seemingly identical files can then be compared in full to check if they are really identical. Using SHA512 would make the process slower, and since we compare files in full anyway there is no risk in a potential false positive from MD5.



When I put my benign software package and my erotic sushi package, which collide under MD5, in your directory, your duplicate-detection script will initially think they are duplicates. In this case, you absolutely must compare the files in full. But there are much better ways to do this!



  • If you use SHA-512, you can safely skip the comparison step. Same if you use BLAKE2b, which can be even faster than MD5.


  • You could even use MD5 safely for this if you use it as HMAC-MD5 under a uniform random key, and safely skip the comparison step. HMAC-MD5 does not seem to be broken, as a pseudorandom function family—so it's probably fine for security, up to the birthday bound, but there are better faster PRFs like keyed BLAKE2 that won't raise any auditors' eyebrows.


  • Even better, you can choose a random key and hash the files with a universal hash under the key, like Poly1305. This is many times faster than MD5 or BLAKE2b, and the probability of a collision between any two files is less than $1/2^100$, so the probability of collision among $n$ files is less than $binom n 2 2^-100$ and thus you can still safely skip the comparison step until you have quadrillions of files.


  • You could also just use a cheap checksum like a CRC with a fixed polynomial. This will be the fastest of the options—far and away faster than MD5—but unlike the previous options you still absolutely must compare the files in full.


So, is MD5 safe for finding candidate duplicates to verify, if you subsequently compare the files bit by bit in full? Yes. So is the constant zero function.




(In a way, this would be creating a rainbow table where all the files are the dictionary)




This is not a rainbow table. A rainbow table is a specific technique for precomputing a random walk over a space of, say, passwords, via, say, MD5 hashes, in a way that saves effort trying to find MD5 preimages for hashes that aren't necessarily in your table in the first place, or doing it in parallel to speed up a multi-target search. It is not simply a list of precomputed hashes on a dictionary of inputs.



(The blog post by tptacek that you cited, and the blog post by Jeff Atwood that it was a response to, are both confused about what rainbow tables are.)




When the password scheme article states that "MD5 is fast", it clearly refers to the problem that hashing MD5 is too cheap when it comes to hashing a large amount of passwords to find the reverse of a hash. But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?




I don't know what tptacek meant—you could email and ask—but if I had to guess, I would guess this meant it's awfully slow for things like hash tables, where you would truncate MD5 to a few bits to determine an index into an array of buckets or an open-addressing array.






share|improve this answer











$endgroup$




















    2












    $begingroup$


    But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?




    BLAKE2 is faster than MD5 and currently known to provide 64-bit collision resistence when truncated to the same size as MD5 (compare ~30 of that of MD5).






    share|improve this answer









    $endgroup$




















      2












      $begingroup$

      There's not a compelling reason to use MD5; however, there are some embedded systems with a MD5 core that was used as a stream verifier. In those systems, MD5 is still used. They are moving to BLAKE2 because it's smaller in silicon, and it has the benefit of being faster than MD5 in general.



      The reason that MD5 started fall out of favor with hardware people was that the word reordering of the MD5 message expansions seems to be simple, but actually
      they require a lot of circuits for demultiplexing and interconnect, and the hardware efficiencies are greatly degraded compared to BLAKE. In contrast, the message expansion blocks for BLAKE algorithms can be efficiently implemented as simple feedback shift registers.



      The BLAKE team did a nice job of making it work well on silicon and in instructions.






      share|improve this answer









      $endgroup$




















        0












        $begingroup$

        A case where the use of the MD5-hash would still make sense (and low risk of deleting duplicated files):



        If you want to find duplicate files you can just use CRC32.



        As soon as two files return the same CRC32-hash you recompute the files with MD5 hash. If the MD5 hash is again identical for both files then you know that the files are duplicates.




        In a case of high risk by deleting files:



        You want the process to be fast: Instead use a hash function that's not vulnerable for a second hash of the files, i.e. SHA2 or SHA3. It's extremely unlikely that these hashes would return an identical hash.



        Speed is no concern: Compare the files byte per byte.






        share|improve this answer











        $endgroup$












        • $begingroup$
          Why use MD5 as a second step after CRC32 instead of SHA2/3 directly? It should make a negligible difference since, absent deliberate file corruption, CRC32 collisions should still be rare. And if you are concerned about deliberate file corruption, you don't use CRC32 nor MD5 in the first place.
          $endgroup$
          – fkraiem
          6 hours ago






        • 6




          $begingroup$
          Why use a second step after CRC32 at all? Compare the files byte-by-byte if you're going to read them again completely anyhow!
          $endgroup$
          – Ruben De Smet
          6 hours ago






        • 1




          $begingroup$
          @RubenDeSmet I think it's because to compare them byte-by-byte you'd have to buffer both files to a certain limit (because of memory constraints) and compare those. This will slow down sequential read speeds because you need to jump between the files. If this actually makes any real world difference provided a large enough buffer size is beyond my knowledge.
          $endgroup$
          – JensV
          2 hours ago






        • 1




          $begingroup$
          @JensV I am pretty sure that the speed difference between a byte-by-byte comparison and a SHA3 comparison (with reasonable buffer sizes) will be trivial. It might even favour the byte-by-byte comparison.
          $endgroup$
          – Martin Bonner
          1 hour ago






        • 3




          $begingroup$
          Comparing the files byte-by-byte requires communication. Computing a hash can be done locally. If the connection is slow compared to the hard drive speed, computing another hash after CRC32 might still be a reasonable option before comparing byte-by-byte.
          $endgroup$
          – JiK
          1 hour ago


















        0












        $begingroup$

        MD5 is currently used throughout the world both at home and in the enterprise. It's the file change mechanism within *nix's rsync if you opt for something other than changed timestamp detection. It's used for backup, archiving and file transfer between in-house systems. Even between enterprises over VPNs.



        Your comment that it "should not be used for integrity checking of documents" is interesting, as that's kinda what is done when transferring files (aka documents). A hacked file/document is philosophically a changed file/document. Yet if the adversary takes advantage of MD5's low collision resistance, file/document propagation through systems can be stopped. Carefully made changes can therefore go unnoticed by rsync, and attacks can occur. I expect that this is somewhat of a niche attack but illustrates the concept vis-à-vis MD5 being at the heart of many computer systems.



        In rsync's case, swapping out MD5 to something faster would only produce marginal overall speed improvement given storage and networking overheads. It would certainly be less than the simple ratio of hash rates suggests.






        share|improve this answer











        $endgroup$













          Your Answer








          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "281"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: false,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          noCode: true, onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













          draft saved

          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcrypto.stackexchange.com%2fquestions%2f70036%2fis-there-really-no-use-for-md5-anymore%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown

























          5 Answers
          5






          active

          oldest

          votes








          5 Answers
          5






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          5












          $begingroup$


          I know that MD5 should not be used for password hashing, and that it also should not be used for integrity checking of documents. There are way too many sources citing MD5 preimaging attacks and MD5s low computation time.




          There is no published preimage attack on MD5 that is cheaper than a generic attack on any 128-bit hash function. But you shouldn't rely on that alone when making security decisions, because cryptography is tricky and adversaries are clever and resourceful and can find ways around it!




          1. Identifying malicious files, such as when Linux Mint's download servers were compromised and an ISO file was replaced by a malicious one; in this case you want to be sure that your file doesn't match; collision attacks aren't a vector here.



          There are two issues here:




          1. If you got the MD5 hash from the same source as the ISO image, there's nothing that would prevent an adversary from replacing both the MD5 hash and the ISO image.



            To prevent this, you and the Linux Mint curators need two channels: one for the hashes which can't be compromised (but need only have very low bandwidth), and another for the ISO image (which needs high bandwidth) on which you can then use the MD5 hash in an attempt to detect compromise.



            There's another way to prevent this: Instead of using the uncompromised channel for the hash of every ISO image over and over again as time goes on—which means more and more opportunities for an attacker to subvert it—use it once initially for a public key, which is then used to sign the ISO images; then there's only one opportunity for an attacker to subvert the public key channel.




          2. Collision attacks may still be a vector in cases like this. Consider the following scenario:



            • I am an evil developer. I write two software packages, whose distributions collide under MD5. One of the packages is benign and will survive review and audit. The other one will surreptitiously replace your family photo album by erotic photographs of sushi.

            • The Linux Mint curators carefully scrutinize and audit everything they publish in their package repository and publish the MD5 hashes of what they have audited in a public place that I can't compromise.

            • The Linux Mint curators cavalierly administer the package distributions in their package repository, under the false impression that the published MD5 hashes will protect users.

            In this scenario, I can replace the benign package by the erotic sushi package, pass the MD5 verification with flying colors, and give you a nasty—and luscious—surprise when you try to look up photos of that old hiking trip you took your kids on.




          1. Finding duplicate files. By MD5-summing all files in a directory structure it's easy to find identical hashes. The seemingly identical files can then be compared in full to check if they are really identical. Using SHA512 would make the process slower, and since we compare files in full anyway there is no risk in a potential false positive from MD5.



          When I put my benign software package and my erotic sushi package, which collide under MD5, in your directory, your duplicate-detection script will initially think they are duplicates. In this case, you absolutely must compare the files in full. But there are much better ways to do this!



          • If you use SHA-512, you can safely skip the comparison step. Same if you use BLAKE2b, which can be even faster than MD5.


          • You could even use MD5 safely for this if you use it as HMAC-MD5 under a uniform random key, and safely skip the comparison step. HMAC-MD5 does not seem to be broken, as a pseudorandom function family—so it's probably fine for security, up to the birthday bound, but there are better faster PRFs like keyed BLAKE2 that won't raise any auditors' eyebrows.


          • Even better, you can choose a random key and hash the files with a universal hash under the key, like Poly1305. This is many times faster than MD5 or BLAKE2b, and the probability of a collision between any two files is less than $1/2^100$, so the probability of collision among $n$ files is less than $binom n 2 2^-100$ and thus you can still safely skip the comparison step until you have quadrillions of files.


          • You could also just use a cheap checksum like a CRC with a fixed polynomial. This will be the fastest of the options—far and away faster than MD5—but unlike the previous options you still absolutely must compare the files in full.


          So, is MD5 safe for finding candidate duplicates to verify, if you subsequently compare the files bit by bit in full? Yes. So is the constant zero function.




          (In a way, this would be creating a rainbow table where all the files are the dictionary)




          This is not a rainbow table. A rainbow table is a specific technique for precomputing a random walk over a space of, say, passwords, via, say, MD5 hashes, in a way that saves effort trying to find MD5 preimages for hashes that aren't necessarily in your table in the first place, or doing it in parallel to speed up a multi-target search. It is not simply a list of precomputed hashes on a dictionary of inputs.



          (The blog post by tptacek that you cited, and the blog post by Jeff Atwood that it was a response to, are both confused about what rainbow tables are.)




          When the password scheme article states that "MD5 is fast", it clearly refers to the problem that hashing MD5 is too cheap when it comes to hashing a large amount of passwords to find the reverse of a hash. But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?




          I don't know what tptacek meant—you could email and ask—but if I had to guess, I would guess this meant it's awfully slow for things like hash tables, where you would truncate MD5 to a few bits to determine an index into an array of buckets or an open-addressing array.






          share|improve this answer











          $endgroup$

















            5












            $begingroup$


            I know that MD5 should not be used for password hashing, and that it also should not be used for integrity checking of documents. There are way too many sources citing MD5 preimaging attacks and MD5s low computation time.




            There is no published preimage attack on MD5 that is cheaper than a generic attack on any 128-bit hash function. But you shouldn't rely on that alone when making security decisions, because cryptography is tricky and adversaries are clever and resourceful and can find ways around it!




            1. Identifying malicious files, such as when Linux Mint's download servers were compromised and an ISO file was replaced by a malicious one; in this case you want to be sure that your file doesn't match; collision attacks aren't a vector here.



            There are two issues here:




            1. If you got the MD5 hash from the same source as the ISO image, there's nothing that would prevent an adversary from replacing both the MD5 hash and the ISO image.



              To prevent this, you and the Linux Mint curators need two channels: one for the hashes which can't be compromised (but need only have very low bandwidth), and another for the ISO image (which needs high bandwidth) on which you can then use the MD5 hash in an attempt to detect compromise.



              There's another way to prevent this: Instead of using the uncompromised channel for the hash of every ISO image over and over again as time goes on—which means more and more opportunities for an attacker to subvert it—use it once initially for a public key, which is then used to sign the ISO images; then there's only one opportunity for an attacker to subvert the public key channel.




            2. Collision attacks may still be a vector in cases like this. Consider the following scenario:



              • I am an evil developer. I write two software packages, whose distributions collide under MD5. One of the packages is benign and will survive review and audit. The other one will surreptitiously replace your family photo album by erotic photographs of sushi.

              • The Linux Mint curators carefully scrutinize and audit everything they publish in their package repository and publish the MD5 hashes of what they have audited in a public place that I can't compromise.

              • The Linux Mint curators cavalierly administer the package distributions in their package repository, under the false impression that the published MD5 hashes will protect users.

              In this scenario, I can replace the benign package by the erotic sushi package, pass the MD5 verification with flying colors, and give you a nasty—and luscious—surprise when you try to look up photos of that old hiking trip you took your kids on.




            1. Finding duplicate files. By MD5-summing all files in a directory structure it's easy to find identical hashes. The seemingly identical files can then be compared in full to check if they are really identical. Using SHA512 would make the process slower, and since we compare files in full anyway there is no risk in a potential false positive from MD5.



            When I put my benign software package and my erotic sushi package, which collide under MD5, in your directory, your duplicate-detection script will initially think they are duplicates. In this case, you absolutely must compare the files in full. But there are much better ways to do this!



            • If you use SHA-512, you can safely skip the comparison step. Same if you use BLAKE2b, which can be even faster than MD5.


            • You could even use MD5 safely for this if you use it as HMAC-MD5 under a uniform random key, and safely skip the comparison step. HMAC-MD5 does not seem to be broken, as a pseudorandom function family—so it's probably fine for security, up to the birthday bound, but there are better faster PRFs like keyed BLAKE2 that won't raise any auditors' eyebrows.


            • Even better, you can choose a random key and hash the files with a universal hash under the key, like Poly1305. This is many times faster than MD5 or BLAKE2b, and the probability of a collision between any two files is less than $1/2^100$, so the probability of collision among $n$ files is less than $binom n 2 2^-100$ and thus you can still safely skip the comparison step until you have quadrillions of files.


            • You could also just use a cheap checksum like a CRC with a fixed polynomial. This will be the fastest of the options—far and away faster than MD5—but unlike the previous options you still absolutely must compare the files in full.


            So, is MD5 safe for finding candidate duplicates to verify, if you subsequently compare the files bit by bit in full? Yes. So is the constant zero function.




            (In a way, this would be creating a rainbow table where all the files are the dictionary)




            This is not a rainbow table. A rainbow table is a specific technique for precomputing a random walk over a space of, say, passwords, via, say, MD5 hashes, in a way that saves effort trying to find MD5 preimages for hashes that aren't necessarily in your table in the first place, or doing it in parallel to speed up a multi-target search. It is not simply a list of precomputed hashes on a dictionary of inputs.



            (The blog post by tptacek that you cited, and the blog post by Jeff Atwood that it was a response to, are both confused about what rainbow tables are.)




            When the password scheme article states that "MD5 is fast", it clearly refers to the problem that hashing MD5 is too cheap when it comes to hashing a large amount of passwords to find the reverse of a hash. But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?




            I don't know what tptacek meant—you could email and ask—but if I had to guess, I would guess this meant it's awfully slow for things like hash tables, where you would truncate MD5 to a few bits to determine an index into an array of buckets or an open-addressing array.






            share|improve this answer











            $endgroup$















              5












              5








              5





              $begingroup$


              I know that MD5 should not be used for password hashing, and that it also should not be used for integrity checking of documents. There are way too many sources citing MD5 preimaging attacks and MD5s low computation time.




              There is no published preimage attack on MD5 that is cheaper than a generic attack on any 128-bit hash function. But you shouldn't rely on that alone when making security decisions, because cryptography is tricky and adversaries are clever and resourceful and can find ways around it!




              1. Identifying malicious files, such as when Linux Mint's download servers were compromised and an ISO file was replaced by a malicious one; in this case you want to be sure that your file doesn't match; collision attacks aren't a vector here.



              There are two issues here:




              1. If you got the MD5 hash from the same source as the ISO image, there's nothing that would prevent an adversary from replacing both the MD5 hash and the ISO image.



                To prevent this, you and the Linux Mint curators need two channels: one for the hashes which can't be compromised (but need only have very low bandwidth), and another for the ISO image (which needs high bandwidth) on which you can then use the MD5 hash in an attempt to detect compromise.



                There's another way to prevent this: Instead of using the uncompromised channel for the hash of every ISO image over and over again as time goes on—which means more and more opportunities for an attacker to subvert it—use it once initially for a public key, which is then used to sign the ISO images; then there's only one opportunity for an attacker to subvert the public key channel.




              2. Collision attacks may still be a vector in cases like this. Consider the following scenario:



                • I am an evil developer. I write two software packages, whose distributions collide under MD5. One of the packages is benign and will survive review and audit. The other one will surreptitiously replace your family photo album by erotic photographs of sushi.

                • The Linux Mint curators carefully scrutinize and audit everything they publish in their package repository and publish the MD5 hashes of what they have audited in a public place that I can't compromise.

                • The Linux Mint curators cavalierly administer the package distributions in their package repository, under the false impression that the published MD5 hashes will protect users.

                In this scenario, I can replace the benign package by the erotic sushi package, pass the MD5 verification with flying colors, and give you a nasty—and luscious—surprise when you try to look up photos of that old hiking trip you took your kids on.




              1. Finding duplicate files. By MD5-summing all files in a directory structure it's easy to find identical hashes. The seemingly identical files can then be compared in full to check if they are really identical. Using SHA512 would make the process slower, and since we compare files in full anyway there is no risk in a potential false positive from MD5.



              When I put my benign software package and my erotic sushi package, which collide under MD5, in your directory, your duplicate-detection script will initially think they are duplicates. In this case, you absolutely must compare the files in full. But there are much better ways to do this!



              • If you use SHA-512, you can safely skip the comparison step. Same if you use BLAKE2b, which can be even faster than MD5.


              • You could even use MD5 safely for this if you use it as HMAC-MD5 under a uniform random key, and safely skip the comparison step. HMAC-MD5 does not seem to be broken, as a pseudorandom function family—so it's probably fine for security, up to the birthday bound, but there are better faster PRFs like keyed BLAKE2 that won't raise any auditors' eyebrows.


              • Even better, you can choose a random key and hash the files with a universal hash under the key, like Poly1305. This is many times faster than MD5 or BLAKE2b, and the probability of a collision between any two files is less than $1/2^100$, so the probability of collision among $n$ files is less than $binom n 2 2^-100$ and thus you can still safely skip the comparison step until you have quadrillions of files.


              • You could also just use a cheap checksum like a CRC with a fixed polynomial. This will be the fastest of the options—far and away faster than MD5—but unlike the previous options you still absolutely must compare the files in full.


              So, is MD5 safe for finding candidate duplicates to verify, if you subsequently compare the files bit by bit in full? Yes. So is the constant zero function.




              (In a way, this would be creating a rainbow table where all the files are the dictionary)




              This is not a rainbow table. A rainbow table is a specific technique for precomputing a random walk over a space of, say, passwords, via, say, MD5 hashes, in a way that saves effort trying to find MD5 preimages for hashes that aren't necessarily in your table in the first place, or doing it in parallel to speed up a multi-target search. It is not simply a list of precomputed hashes on a dictionary of inputs.



              (The blog post by tptacek that you cited, and the blog post by Jeff Atwood that it was a response to, are both confused about what rainbow tables are.)




              When the password scheme article states that "MD5 is fast", it clearly refers to the problem that hashing MD5 is too cheap when it comes to hashing a large amount of passwords to find the reverse of a hash. But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?




              I don't know what tptacek meant—you could email and ask—but if I had to guess, I would guess this meant it's awfully slow for things like hash tables, where you would truncate MD5 to a few bits to determine an index into an array of buckets or an open-addressing array.






              share|improve this answer











              $endgroup$




              I know that MD5 should not be used for password hashing, and that it also should not be used for integrity checking of documents. There are way too many sources citing MD5 preimaging attacks and MD5s low computation time.




              There is no published preimage attack on MD5 that is cheaper than a generic attack on any 128-bit hash function. But you shouldn't rely on that alone when making security decisions, because cryptography is tricky and adversaries are clever and resourceful and can find ways around it!




              1. Identifying malicious files, such as when Linux Mint's download servers were compromised and an ISO file was replaced by a malicious one; in this case you want to be sure that your file doesn't match; collision attacks aren't a vector here.



              There are two issues here:




              1. If you got the MD5 hash from the same source as the ISO image, there's nothing that would prevent an adversary from replacing both the MD5 hash and the ISO image.



                To prevent this, you and the Linux Mint curators need two channels: one for the hashes which can't be compromised (but need only have very low bandwidth), and another for the ISO image (which needs high bandwidth) on which you can then use the MD5 hash in an attempt to detect compromise.



                There's another way to prevent this: Instead of using the uncompromised channel for the hash of every ISO image over and over again as time goes on—which means more and more opportunities for an attacker to subvert it—use it once initially for a public key, which is then used to sign the ISO images; then there's only one opportunity for an attacker to subvert the public key channel.




              2. Collision attacks may still be a vector in cases like this. Consider the following scenario:



                • I am an evil developer. I write two software packages, whose distributions collide under MD5. One of the packages is benign and will survive review and audit. The other one will surreptitiously replace your family photo album by erotic photographs of sushi.

                • The Linux Mint curators carefully scrutinize and audit everything they publish in their package repository and publish the MD5 hashes of what they have audited in a public place that I can't compromise.

                • The Linux Mint curators cavalierly administer the package distributions in their package repository, under the false impression that the published MD5 hashes will protect users.

                In this scenario, I can replace the benign package by the erotic sushi package, pass the MD5 verification with flying colors, and give you a nasty—and luscious—surprise when you try to look up photos of that old hiking trip you took your kids on.




              1. Finding duplicate files. By MD5-summing all files in a directory structure it's easy to find identical hashes. The seemingly identical files can then be compared in full to check if they are really identical. Using SHA512 would make the process slower, and since we compare files in full anyway there is no risk in a potential false positive from MD5.



              When I put my benign software package and my erotic sushi package, which collide under MD5, in your directory, your duplicate-detection script will initially think they are duplicates. In this case, you absolutely must compare the files in full. But there are much better ways to do this!



              • If you use SHA-512, you can safely skip the comparison step. Same if you use BLAKE2b, which can be even faster than MD5.


              • You could even use MD5 safely for this if you use it as HMAC-MD5 under a uniform random key, and safely skip the comparison step. HMAC-MD5 does not seem to be broken, as a pseudorandom function family—so it's probably fine for security, up to the birthday bound, but there are better faster PRFs like keyed BLAKE2 that won't raise any auditors' eyebrows.


              • Even better, you can choose a random key and hash the files with a universal hash under the key, like Poly1305. This is many times faster than MD5 or BLAKE2b, and the probability of a collision between any two files is less than $1/2^100$, so the probability of collision among $n$ files is less than $binom n 2 2^-100$ and thus you can still safely skip the comparison step until you have quadrillions of files.


              • You could also just use a cheap checksum like a CRC with a fixed polynomial. This will be the fastest of the options—far and away faster than MD5—but unlike the previous options you still absolutely must compare the files in full.


              So, is MD5 safe for finding candidate duplicates to verify, if you subsequently compare the files bit by bit in full? Yes. So is the constant zero function.




              (In a way, this would be creating a rainbow table where all the files are the dictionary)




              This is not a rainbow table. A rainbow table is a specific technique for precomputing a random walk over a space of, say, passwords, via, say, MD5 hashes, in a way that saves effort trying to find MD5 preimages for hashes that aren't necessarily in your table in the first place, or doing it in parallel to speed up a multi-target search. It is not simply a list of precomputed hashes on a dictionary of inputs.



              (The blog post by tptacek that you cited, and the blog post by Jeff Atwood that it was a response to, are both confused about what rainbow tables are.)




              When the password scheme article states that "MD5 is fast", it clearly refers to the problem that hashing MD5 is too cheap when it comes to hashing a large amount of passwords to find the reverse of a hash. But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?




              I don't know what tptacek meant—you could email and ask—but if I had to guess, I would guess this meant it's awfully slow for things like hash tables, where you would truncate MD5 to a few bits to determine an index into an array of buckets or an open-addressing array.







              share|improve this answer














              share|improve this answer



              share|improve this answer








              edited 35 mins ago

























              answered 1 hour ago









              Squeamish OssifrageSqueamish Ossifrage

              23.2k133106




              23.2k133106





















                  2












                  $begingroup$


                  But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?




                  BLAKE2 is faster than MD5 and currently known to provide 64-bit collision resistence when truncated to the same size as MD5 (compare ~30 of that of MD5).






                  share|improve this answer









                  $endgroup$

















                    2












                    $begingroup$


                    But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?




                    BLAKE2 is faster than MD5 and currently known to provide 64-bit collision resistence when truncated to the same size as MD5 (compare ~30 of that of MD5).






                    share|improve this answer









                    $endgroup$















                      2












                      2








                      2





                      $begingroup$


                      But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?




                      BLAKE2 is faster than MD5 and currently known to provide 64-bit collision resistence when truncated to the same size as MD5 (compare ~30 of that of MD5).






                      share|improve this answer









                      $endgroup$




                      But what does it mean when it says that "[MD5 is] too slow to use as a general purpose hash"? Are there faster standardized hashes to compare files, that still have a reasonably low chance of collision?




                      BLAKE2 is faster than MD5 and currently known to provide 64-bit collision resistence when truncated to the same size as MD5 (compare ~30 of that of MD5).







                      share|improve this answer












                      share|improve this answer



                      share|improve this answer










                      answered 4 hours ago









                      DannyNiuDannyNiu

                      1,4151629




                      1,4151629





















                          2












                          $begingroup$

                          There's not a compelling reason to use MD5; however, there are some embedded systems with a MD5 core that was used as a stream verifier. In those systems, MD5 is still used. They are moving to BLAKE2 because it's smaller in silicon, and it has the benefit of being faster than MD5 in general.



                          The reason that MD5 started fall out of favor with hardware people was that the word reordering of the MD5 message expansions seems to be simple, but actually
                          they require a lot of circuits for demultiplexing and interconnect, and the hardware efficiencies are greatly degraded compared to BLAKE. In contrast, the message expansion blocks for BLAKE algorithms can be efficiently implemented as simple feedback shift registers.



                          The BLAKE team did a nice job of making it work well on silicon and in instructions.






                          share|improve this answer









                          $endgroup$

















                            2












                            $begingroup$

                            There's not a compelling reason to use MD5; however, there are some embedded systems with a MD5 core that was used as a stream verifier. In those systems, MD5 is still used. They are moving to BLAKE2 because it's smaller in silicon, and it has the benefit of being faster than MD5 in general.



                            The reason that MD5 started fall out of favor with hardware people was that the word reordering of the MD5 message expansions seems to be simple, but actually
                            they require a lot of circuits for demultiplexing and interconnect, and the hardware efficiencies are greatly degraded compared to BLAKE. In contrast, the message expansion blocks for BLAKE algorithms can be efficiently implemented as simple feedback shift registers.



                            The BLAKE team did a nice job of making it work well on silicon and in instructions.






                            share|improve this answer









                            $endgroup$















                              2












                              2








                              2





                              $begingroup$

                              There's not a compelling reason to use MD5; however, there are some embedded systems with a MD5 core that was used as a stream verifier. In those systems, MD5 is still used. They are moving to BLAKE2 because it's smaller in silicon, and it has the benefit of being faster than MD5 in general.



                              The reason that MD5 started fall out of favor with hardware people was that the word reordering of the MD5 message expansions seems to be simple, but actually
                              they require a lot of circuits for demultiplexing and interconnect, and the hardware efficiencies are greatly degraded compared to BLAKE. In contrast, the message expansion blocks for BLAKE algorithms can be efficiently implemented as simple feedback shift registers.



                              The BLAKE team did a nice job of making it work well on silicon and in instructions.






                              share|improve this answer









                              $endgroup$



                              There's not a compelling reason to use MD5; however, there are some embedded systems with a MD5 core that was used as a stream verifier. In those systems, MD5 is still used. They are moving to BLAKE2 because it's smaller in silicon, and it has the benefit of being faster than MD5 in general.



                              The reason that MD5 started fall out of favor with hardware people was that the word reordering of the MD5 message expansions seems to be simple, but actually
                              they require a lot of circuits for demultiplexing and interconnect, and the hardware efficiencies are greatly degraded compared to BLAKE. In contrast, the message expansion blocks for BLAKE algorithms can be efficiently implemented as simple feedback shift registers.



                              The BLAKE team did a nice job of making it work well on silicon and in instructions.







                              share|improve this answer












                              share|improve this answer



                              share|improve this answer










                              answered 2 hours ago









                              b degnanb degnan

                              2,1021829




                              2,1021829





















                                  0












                                  $begingroup$

                                  A case where the use of the MD5-hash would still make sense (and low risk of deleting duplicated files):



                                  If you want to find duplicate files you can just use CRC32.



                                  As soon as two files return the same CRC32-hash you recompute the files with MD5 hash. If the MD5 hash is again identical for both files then you know that the files are duplicates.




                                  In a case of high risk by deleting files:



                                  You want the process to be fast: Instead use a hash function that's not vulnerable for a second hash of the files, i.e. SHA2 or SHA3. It's extremely unlikely that these hashes would return an identical hash.



                                  Speed is no concern: Compare the files byte per byte.






                                  share|improve this answer











                                  $endgroup$












                                  • $begingroup$
                                    Why use MD5 as a second step after CRC32 instead of SHA2/3 directly? It should make a negligible difference since, absent deliberate file corruption, CRC32 collisions should still be rare. And if you are concerned about deliberate file corruption, you don't use CRC32 nor MD5 in the first place.
                                    $endgroup$
                                    – fkraiem
                                    6 hours ago






                                  • 6




                                    $begingroup$
                                    Why use a second step after CRC32 at all? Compare the files byte-by-byte if you're going to read them again completely anyhow!
                                    $endgroup$
                                    – Ruben De Smet
                                    6 hours ago






                                  • 1




                                    $begingroup$
                                    @RubenDeSmet I think it's because to compare them byte-by-byte you'd have to buffer both files to a certain limit (because of memory constraints) and compare those. This will slow down sequential read speeds because you need to jump between the files. If this actually makes any real world difference provided a large enough buffer size is beyond my knowledge.
                                    $endgroup$
                                    – JensV
                                    2 hours ago






                                  • 1




                                    $begingroup$
                                    @JensV I am pretty sure that the speed difference between a byte-by-byte comparison and a SHA3 comparison (with reasonable buffer sizes) will be trivial. It might even favour the byte-by-byte comparison.
                                    $endgroup$
                                    – Martin Bonner
                                    1 hour ago






                                  • 3




                                    $begingroup$
                                    Comparing the files byte-by-byte requires communication. Computing a hash can be done locally. If the connection is slow compared to the hard drive speed, computing another hash after CRC32 might still be a reasonable option before comparing byte-by-byte.
                                    $endgroup$
                                    – JiK
                                    1 hour ago















                                  0












                                  $begingroup$

                                  A case where the use of the MD5-hash would still make sense (and low risk of deleting duplicated files):



                                  If you want to find duplicate files you can just use CRC32.



                                  As soon as two files return the same CRC32-hash you recompute the files with MD5 hash. If the MD5 hash is again identical for both files then you know that the files are duplicates.




                                  In a case of high risk by deleting files:



                                  You want the process to be fast: Instead use a hash function that's not vulnerable for a second hash of the files, i.e. SHA2 or SHA3. It's extremely unlikely that these hashes would return an identical hash.



                                  Speed is no concern: Compare the files byte per byte.






                                  share|improve this answer











                                  $endgroup$












                                  • $begingroup$
                                    Why use MD5 as a second step after CRC32 instead of SHA2/3 directly? It should make a negligible difference since, absent deliberate file corruption, CRC32 collisions should still be rare. And if you are concerned about deliberate file corruption, you don't use CRC32 nor MD5 in the first place.
                                    $endgroup$
                                    – fkraiem
                                    6 hours ago






                                  • 6




                                    $begingroup$
                                    Why use a second step after CRC32 at all? Compare the files byte-by-byte if you're going to read them again completely anyhow!
                                    $endgroup$
                                    – Ruben De Smet
                                    6 hours ago






                                  • 1




                                    $begingroup$
                                    @RubenDeSmet I think it's because to compare them byte-by-byte you'd have to buffer both files to a certain limit (because of memory constraints) and compare those. This will slow down sequential read speeds because you need to jump between the files. If this actually makes any real world difference provided a large enough buffer size is beyond my knowledge.
                                    $endgroup$
                                    – JensV
                                    2 hours ago






                                  • 1




                                    $begingroup$
                                    @JensV I am pretty sure that the speed difference between a byte-by-byte comparison and a SHA3 comparison (with reasonable buffer sizes) will be trivial. It might even favour the byte-by-byte comparison.
                                    $endgroup$
                                    – Martin Bonner
                                    1 hour ago






                                  • 3




                                    $begingroup$
                                    Comparing the files byte-by-byte requires communication. Computing a hash can be done locally. If the connection is slow compared to the hard drive speed, computing another hash after CRC32 might still be a reasonable option before comparing byte-by-byte.
                                    $endgroup$
                                    – JiK
                                    1 hour ago













                                  0












                                  0








                                  0





                                  $begingroup$

                                  A case where the use of the MD5-hash would still make sense (and low risk of deleting duplicated files):



                                  If you want to find duplicate files you can just use CRC32.



                                  As soon as two files return the same CRC32-hash you recompute the files with MD5 hash. If the MD5 hash is again identical for both files then you know that the files are duplicates.




                                  In a case of high risk by deleting files:



                                  You want the process to be fast: Instead use a hash function that's not vulnerable for a second hash of the files, i.e. SHA2 or SHA3. It's extremely unlikely that these hashes would return an identical hash.



                                  Speed is no concern: Compare the files byte per byte.






                                  share|improve this answer











                                  $endgroup$



                                  A case where the use of the MD5-hash would still make sense (and low risk of deleting duplicated files):



                                  If you want to find duplicate files you can just use CRC32.



                                  As soon as two files return the same CRC32-hash you recompute the files with MD5 hash. If the MD5 hash is again identical for both files then you know that the files are duplicates.




                                  In a case of high risk by deleting files:



                                  You want the process to be fast: Instead use a hash function that's not vulnerable for a second hash of the files, i.e. SHA2 or SHA3. It's extremely unlikely that these hashes would return an identical hash.



                                  Speed is no concern: Compare the files byte per byte.







                                  share|improve this answer














                                  share|improve this answer



                                  share|improve this answer








                                  edited 6 hours ago

























                                  answered 7 hours ago









                                  AleksanderRasAleksanderRas

                                  3,1091937




                                  3,1091937











                                  • $begingroup$
                                    Why use MD5 as a second step after CRC32 instead of SHA2/3 directly? It should make a negligible difference since, absent deliberate file corruption, CRC32 collisions should still be rare. And if you are concerned about deliberate file corruption, you don't use CRC32 nor MD5 in the first place.
                                    $endgroup$
                                    – fkraiem
                                    6 hours ago






                                  • 6




                                    $begingroup$
                                    Why use a second step after CRC32 at all? Compare the files byte-by-byte if you're going to read them again completely anyhow!
                                    $endgroup$
                                    – Ruben De Smet
                                    6 hours ago






                                  • 1




                                    $begingroup$
                                    @RubenDeSmet I think it's because to compare them byte-by-byte you'd have to buffer both files to a certain limit (because of memory constraints) and compare those. This will slow down sequential read speeds because you need to jump between the files. If this actually makes any real world difference provided a large enough buffer size is beyond my knowledge.
                                    $endgroup$
                                    – JensV
                                    2 hours ago






                                  • 1




                                    $begingroup$
                                    @JensV I am pretty sure that the speed difference between a byte-by-byte comparison and a SHA3 comparison (with reasonable buffer sizes) will be trivial. It might even favour the byte-by-byte comparison.
                                    $endgroup$
                                    – Martin Bonner
                                    1 hour ago






                                  • 3




                                    $begingroup$
                                    Comparing the files byte-by-byte requires communication. Computing a hash can be done locally. If the connection is slow compared to the hard drive speed, computing another hash after CRC32 might still be a reasonable option before comparing byte-by-byte.
                                    $endgroup$
                                    – JiK
                                    1 hour ago
















                                  • $begingroup$
                                    Why use MD5 as a second step after CRC32 instead of SHA2/3 directly? It should make a negligible difference since, absent deliberate file corruption, CRC32 collisions should still be rare. And if you are concerned about deliberate file corruption, you don't use CRC32 nor MD5 in the first place.
                                    $endgroup$
                                    – fkraiem
                                    6 hours ago






                                  • 6




                                    $begingroup$
                                    Why use a second step after CRC32 at all? Compare the files byte-by-byte if you're going to read them again completely anyhow!
                                    $endgroup$
                                    – Ruben De Smet
                                    6 hours ago






                                  • 1




                                    $begingroup$
                                    @RubenDeSmet I think it's because to compare them byte-by-byte you'd have to buffer both files to a certain limit (because of memory constraints) and compare those. This will slow down sequential read speeds because you need to jump between the files. If this actually makes any real world difference provided a large enough buffer size is beyond my knowledge.
                                    $endgroup$
                                    – JensV
                                    2 hours ago






                                  • 1




                                    $begingroup$
                                    @JensV I am pretty sure that the speed difference between a byte-by-byte comparison and a SHA3 comparison (with reasonable buffer sizes) will be trivial. It might even favour the byte-by-byte comparison.
                                    $endgroup$
                                    – Martin Bonner
                                    1 hour ago






                                  • 3




                                    $begingroup$
                                    Comparing the files byte-by-byte requires communication. Computing a hash can be done locally. If the connection is slow compared to the hard drive speed, computing another hash after CRC32 might still be a reasonable option before comparing byte-by-byte.
                                    $endgroup$
                                    – JiK
                                    1 hour ago















                                  $begingroup$
                                  Why use MD5 as a second step after CRC32 instead of SHA2/3 directly? It should make a negligible difference since, absent deliberate file corruption, CRC32 collisions should still be rare. And if you are concerned about deliberate file corruption, you don't use CRC32 nor MD5 in the first place.
                                  $endgroup$
                                  – fkraiem
                                  6 hours ago




                                  $begingroup$
                                  Why use MD5 as a second step after CRC32 instead of SHA2/3 directly? It should make a negligible difference since, absent deliberate file corruption, CRC32 collisions should still be rare. And if you are concerned about deliberate file corruption, you don't use CRC32 nor MD5 in the first place.
                                  $endgroup$
                                  – fkraiem
                                  6 hours ago




                                  6




                                  6




                                  $begingroup$
                                  Why use a second step after CRC32 at all? Compare the files byte-by-byte if you're going to read them again completely anyhow!
                                  $endgroup$
                                  – Ruben De Smet
                                  6 hours ago




                                  $begingroup$
                                  Why use a second step after CRC32 at all? Compare the files byte-by-byte if you're going to read them again completely anyhow!
                                  $endgroup$
                                  – Ruben De Smet
                                  6 hours ago




                                  1




                                  1




                                  $begingroup$
                                  @RubenDeSmet I think it's because to compare them byte-by-byte you'd have to buffer both files to a certain limit (because of memory constraints) and compare those. This will slow down sequential read speeds because you need to jump between the files. If this actually makes any real world difference provided a large enough buffer size is beyond my knowledge.
                                  $endgroup$
                                  – JensV
                                  2 hours ago




                                  $begingroup$
                                  @RubenDeSmet I think it's because to compare them byte-by-byte you'd have to buffer both files to a certain limit (because of memory constraints) and compare those. This will slow down sequential read speeds because you need to jump between the files. If this actually makes any real world difference provided a large enough buffer size is beyond my knowledge.
                                  $endgroup$
                                  – JensV
                                  2 hours ago




                                  1




                                  1




                                  $begingroup$
                                  @JensV I am pretty sure that the speed difference between a byte-by-byte comparison and a SHA3 comparison (with reasonable buffer sizes) will be trivial. It might even favour the byte-by-byte comparison.
                                  $endgroup$
                                  – Martin Bonner
                                  1 hour ago




                                  $begingroup$
                                  @JensV I am pretty sure that the speed difference between a byte-by-byte comparison and a SHA3 comparison (with reasonable buffer sizes) will be trivial. It might even favour the byte-by-byte comparison.
                                  $endgroup$
                                  – Martin Bonner
                                  1 hour ago




                                  3




                                  3




                                  $begingroup$
                                  Comparing the files byte-by-byte requires communication. Computing a hash can be done locally. If the connection is slow compared to the hard drive speed, computing another hash after CRC32 might still be a reasonable option before comparing byte-by-byte.
                                  $endgroup$
                                  – JiK
                                  1 hour ago




                                  $begingroup$
                                  Comparing the files byte-by-byte requires communication. Computing a hash can be done locally. If the connection is slow compared to the hard drive speed, computing another hash after CRC32 might still be a reasonable option before comparing byte-by-byte.
                                  $endgroup$
                                  – JiK
                                  1 hour ago











                                  0












                                  $begingroup$

                                  MD5 is currently used throughout the world both at home and in the enterprise. It's the file change mechanism within *nix's rsync if you opt for something other than changed timestamp detection. It's used for backup, archiving and file transfer between in-house systems. Even between enterprises over VPNs.



                                  Your comment that it "should not be used for integrity checking of documents" is interesting, as that's kinda what is done when transferring files (aka documents). A hacked file/document is philosophically a changed file/document. Yet if the adversary takes advantage of MD5's low collision resistance, file/document propagation through systems can be stopped. Carefully made changes can therefore go unnoticed by rsync, and attacks can occur. I expect that this is somewhat of a niche attack but illustrates the concept vis-à-vis MD5 being at the heart of many computer systems.



                                  In rsync's case, swapping out MD5 to something faster would only produce marginal overall speed improvement given storage and networking overheads. It would certainly be less than the simple ratio of hash rates suggests.






                                  share|improve this answer











                                  $endgroup$

















                                    0












                                    $begingroup$

                                    MD5 is currently used throughout the world both at home and in the enterprise. It's the file change mechanism within *nix's rsync if you opt for something other than changed timestamp detection. It's used for backup, archiving and file transfer between in-house systems. Even between enterprises over VPNs.



                                    Your comment that it "should not be used for integrity checking of documents" is interesting, as that's kinda what is done when transferring files (aka documents). A hacked file/document is philosophically a changed file/document. Yet if the adversary takes advantage of MD5's low collision resistance, file/document propagation through systems can be stopped. Carefully made changes can therefore go unnoticed by rsync, and attacks can occur. I expect that this is somewhat of a niche attack but illustrates the concept vis-à-vis MD5 being at the heart of many computer systems.



                                    In rsync's case, swapping out MD5 to something faster would only produce marginal overall speed improvement given storage and networking overheads. It would certainly be less than the simple ratio of hash rates suggests.






                                    share|improve this answer











                                    $endgroup$















                                      0












                                      0








                                      0





                                      $begingroup$

                                      MD5 is currently used throughout the world both at home and in the enterprise. It's the file change mechanism within *nix's rsync if you opt for something other than changed timestamp detection. It's used for backup, archiving and file transfer between in-house systems. Even between enterprises over VPNs.



                                      Your comment that it "should not be used for integrity checking of documents" is interesting, as that's kinda what is done when transferring files (aka documents). A hacked file/document is philosophically a changed file/document. Yet if the adversary takes advantage of MD5's low collision resistance, file/document propagation through systems can be stopped. Carefully made changes can therefore go unnoticed by rsync, and attacks can occur. I expect that this is somewhat of a niche attack but illustrates the concept vis-à-vis MD5 being at the heart of many computer systems.



                                      In rsync's case, swapping out MD5 to something faster would only produce marginal overall speed improvement given storage and networking overheads. It would certainly be less than the simple ratio of hash rates suggests.






                                      share|improve this answer











                                      $endgroup$



                                      MD5 is currently used throughout the world both at home and in the enterprise. It's the file change mechanism within *nix's rsync if you opt for something other than changed timestamp detection. It's used for backup, archiving and file transfer between in-house systems. Even between enterprises over VPNs.



                                      Your comment that it "should not be used for integrity checking of documents" is interesting, as that's kinda what is done when transferring files (aka documents). A hacked file/document is philosophically a changed file/document. Yet if the adversary takes advantage of MD5's low collision resistance, file/document propagation through systems can be stopped. Carefully made changes can therefore go unnoticed by rsync, and attacks can occur. I expect that this is somewhat of a niche attack but illustrates the concept vis-à-vis MD5 being at the heart of many computer systems.



                                      In rsync's case, swapping out MD5 to something faster would only produce marginal overall speed improvement given storage and networking overheads. It would certainly be less than the simple ratio of hash rates suggests.







                                      share|improve this answer














                                      share|improve this answer



                                      share|improve this answer








                                      edited 13 mins ago

























                                      answered 3 hours ago









                                      Paul UszakPaul Uszak

                                      7,80011638




                                      7,80011638



























                                          draft saved

                                          draft discarded
















































                                          Thanks for contributing an answer to Cryptography Stack Exchange!


                                          • Please be sure to answer the question. Provide details and share your research!

                                          But avoid


                                          • Asking for help, clarification, or responding to other answers.

                                          • Making statements based on opinion; back them up with references or personal experience.

                                          Use MathJax to format equations. MathJax reference.


                                          To learn more, see our tips on writing great answers.




                                          draft saved


                                          draft discarded














                                          StackExchange.ready(
                                          function ()
                                          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcrypto.stackexchange.com%2fquestions%2f70036%2fis-there-really-no-use-for-md5-anymore%23new-answer', 'question_page');

                                          );

                                          Post as a guest















                                          Required, but never shown





















































                                          Required, but never shown














                                          Required, but never shown












                                          Required, but never shown







                                          Required, but never shown

































                                          Required, but never shown














                                          Required, but never shown












                                          Required, but never shown







                                          Required, but never shown







                                          Popular posts from this blog

                                          Oświęcim Innehåll Historia | Källor | Externa länkar | Navigeringsmeny50°2′18″N 19°13′17″Ö / 50.03833°N 19.22139°Ö / 50.03833; 19.2213950°2′18″N 19°13′17″Ö / 50.03833°N 19.22139°Ö / 50.03833; 19.221393089658Nordisk familjebok, AuschwitzInsidan tro och existensJewish Community i OświęcimAuschwitz Jewish Center: MuseumAuschwitz Jewish Center

                                          Valle di Casies Indice Geografia fisica | Origini del nome | Storia | Società | Amministrazione | Sport | Note | Bibliografia | Voci correlate | Altri progetti | Collegamenti esterni | Menu di navigazione46°46′N 12°11′E / 46.766667°N 12.183333°E46.766667; 12.183333 (Valle di Casies)46°46′N 12°11′E / 46.766667°N 12.183333°E46.766667; 12.183333 (Valle di Casies)Sito istituzionaleAstat Censimento della popolazione 2011 - Determinazione della consistenza dei tre gruppi linguistici della Provincia Autonoma di Bolzano-Alto Adige - giugno 2012Numeri e fattiValle di CasiesDato IstatTabella dei gradi/giorno dei Comuni italiani raggruppati per Regione e Provincia26 agosto 1993, n. 412Heraldry of the World: GsiesStatistiche I.StatValCasies.comWikimedia CommonsWikimedia CommonsValle di CasiesSito ufficialeValle di CasiesMM14870458910042978-6

                                          Typsetting diagram chases (with TikZ?) Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)How to define the default vertical distance between nodes?Draw edge on arcNumerical conditional within tikz keys?TikZ: Drawing an arc from an intersection to an intersectionDrawing rectilinear curves in Tikz, aka an Etch-a-Sketch drawingLine up nested tikz enviroments or how to get rid of themHow to place nodes in an absolute coordinate system in tikzCommutative diagram with curve connecting between nodesTikz with standalone: pinning tikz coordinates to page cmDrawing a Decision Diagram with Tikz and layout manager