You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

test-pdiff-usage 19 KiB

implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
stop handling items in doomed transactions With the previous commit we track the state of transactions, so we can now use our knowledge to avoid processing data for a transaction which was already closed (via an abort in this case). This is needed as multiple independent processes are interacting in the process, so there isn't a simple immediate full-engine stop and it would also be bad to teach each and every item how to check if its manager has failed subordinate and what to do in that case. In the pdiff case, which deals (potentially) with many items during its lifetime e.g. a hashsum mismatch in another file can abort the transaction the file we try to patch via pdiff belongs to. This causes some of the items (which are already done) to be aborted with it, but items still in the process of acquisition continue in the processing and will later try to use all the items together failing in strange ways as cleanup already happened. The chosen solution is to dry up the communication channels instead by ignoring new requests for data acquisition, canceling requests which are not assigned to a queue and not calling Done/Failed on items anymore. This means that e.g. already started or pending (e.g. pipelined) downloads aren't stopped and continue as normal for now, but they remain in partial/ and aren't processed further so the next update command will pick them up and put them to good use while the current process fails updating (for this transaction group) in an orderly fashion. Closes: 817240 Thanks: Barr Detwix & Vincent Lefevre for log files
5 years ago
stop handling items in doomed transactions With the previous commit we track the state of transactions, so we can now use our knowledge to avoid processing data for a transaction which was already closed (via an abort in this case). This is needed as multiple independent processes are interacting in the process, so there isn't a simple immediate full-engine stop and it would also be bad to teach each and every item how to check if its manager has failed subordinate and what to do in that case. In the pdiff case, which deals (potentially) with many items during its lifetime e.g. a hashsum mismatch in another file can abort the transaction the file we try to patch via pdiff belongs to. This causes some of the items (which are already done) to be aborted with it, but items still in the process of acquisition continue in the processing and will later try to use all the items together failing in strange ways as cleanup already happened. The chosen solution is to dry up the communication channels instead by ignoring new requests for data acquisition, canceling requests which are not assigned to a queue and not calling Done/Failed on items anymore. This means that e.g. already started or pending (e.g. pipelined) downloads aren't stopped and continue as normal for now, but they remain in partial/ and aren't processed further so the next update command will pick them up and put them to good use while the current process fails updating (for this transaction group) in an orderly fashion. Closes: 817240 Thanks: Barr Detwix & Vincent Lefevre for log files
5 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
stop handling items in doomed transactions With the previous commit we track the state of transactions, so we can now use our knowledge to avoid processing data for a transaction which was already closed (via an abort in this case). This is needed as multiple independent processes are interacting in the process, so there isn't a simple immediate full-engine stop and it would also be bad to teach each and every item how to check if its manager has failed subordinate and what to do in that case. In the pdiff case, which deals (potentially) with many items during its lifetime e.g. a hashsum mismatch in another file can abort the transaction the file we try to patch via pdiff belongs to. This causes some of the items (which are already done) to be aborted with it, but items still in the process of acquisition continue in the processing and will later try to use all the items together failing in strange ways as cleanup already happened. The chosen solution is to dry up the communication channels instead by ignoring new requests for data acquisition, canceling requests which are not assigned to a queue and not calling Done/Failed on items anymore. This means that e.g. already started or pending (e.g. pipelined) downloads aren't stopped and continue as normal for now, but they remain in partial/ and aren't processed further so the next update command will pick them up and put them to good use while the current process fails updating (for this transaction group) in an orderly fashion. Closes: 817240 Thanks: Barr Detwix & Vincent Lefevre for log files
5 years ago
stop handling items in doomed transactions With the previous commit we track the state of transactions, so we can now use our knowledge to avoid processing data for a transaction which was already closed (via an abort in this case). This is needed as multiple independent processes are interacting in the process, so there isn't a simple immediate full-engine stop and it would also be bad to teach each and every item how to check if its manager has failed subordinate and what to do in that case. In the pdiff case, which deals (potentially) with many items during its lifetime e.g. a hashsum mismatch in another file can abort the transaction the file we try to patch via pdiff belongs to. This causes some of the items (which are already done) to be aborted with it, but items still in the process of acquisition continue in the processing and will later try to use all the items together failing in strange ways as cleanup already happened. The chosen solution is to dry up the communication channels instead by ignoring new requests for data acquisition, canceling requests which are not assigned to a queue and not calling Done/Failed on items anymore. This means that e.g. already started or pending (e.g. pipelined) downloads aren't stopped and continue as normal for now, but they remain in partial/ and aren't processed further so the next update command will pick them up and put them to good use while the current process fails updating (for this transaction group) in an orderly fashion. Closes: 817240 Thanks: Barr Detwix & Vincent Lefevre for log files
5 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417
  1. #!/bin/sh
  2. set -e
  3. TESTDIR="$(readlink -f "$(dirname "$0")")"
  4. . "$TESTDIR/framework"
  5. setupenvironment
  6. configarchitecture 'i386'
  7. LOWCOSTEXT='lz4'
  8. buildaptarchive
  9. setupflataptarchive
  10. changetowebserver
  11. cat >rootdir/etc/apt/apt.conf.d/contents.conf <<EOF
  12. Acquire::IndexTargets::deb::Contents {
  13. MetaKey "\$(COMPONENT)/Contents-\$(ARCHITECTURE)";
  14. ShortDescription "Contents";
  15. Description "\$(RELEASE)/\$(COMPONENT) \$(ARCHITECTURE) Contents";
  16. MetaKey "\$(COMPONENT)/Contents-\$(ARCHITECTURE)";
  17. flatMetaKey "Contents-\$(ARCHITECTURE)";
  18. flatDescription "\$(RELEASE) \$(ARCHITECTURE) Contents";
  19. };
  20. EOF
  21. PKGFILE="${TESTDIR}/$(echo "$(basename $0)" | sed 's#^test-#Packages-#')"
  22. echo 'contents for stuff' > aptarchive/Contents-i386
  23. compressfile aptarchive/Contents-i386
  24. echo 'hacked' > aptarchive/hacked-i386
  25. compressfile aptarchive/hacked-i386
  26. wasmergeused() {
  27. if echo "$*" | grep -q -- '-o test::cannot-use-pdiff=1'; then
  28. find rootdir/var/lib/apt/lists/partial -name '*-patched*' -delete
  29. fi
  30. testsuccess apt update "$@"
  31. #apt update "$@" 2>&1 | tee rootdir/tmp/testsuccess.output
  32. msgtest 'No intermediate patch files' 'still exist'
  33. local EDS="$(find rootdir/var/lib/apt/lists -name '*.ed' -o -name '*.ed.*')"
  34. if [ -z "$EDS" ]; then
  35. msgpass
  36. else
  37. echo
  38. echo "$EDS"
  39. msgfail
  40. fi
  41. if echo "$*" | grep -q -- '-o test::cannot-use-pdiff=1'; then
  42. msgtest 'Check if pdiff was' 'not used'
  43. cp -a rootdir/tmp/testsuccess.output rootdir/tmp/aptupdate.output
  44. testsuccess --nomsg grep "^Ign:.* Packages\.diff/Index" rootdir/tmp/aptupdate.output
  45. return;
  46. fi
  47. msgtest 'Check if the right pdiff merger was used'
  48. if grep -q '^pkgAcqIndexMergeDiffs::Done(): rred' rootdir/tmp/testsuccess.output; then
  49. if echo "$*" | grep -q -- '-o Acquire::PDiffs::Merge=1'; then
  50. msgpass
  51. else
  52. msgfail "Merge shouldn't have been used, but was"
  53. fi
  54. elif echo "$*" | grep -q -- '-o Acquire::PDiffs::Merge=1'; then
  55. msgfail "Merge should have been used, but wasn't"
  56. else
  57. msgpass
  58. fi
  59. testequal '1' grep -c rred:601 rootdir/tmp/testsuccess.output
  60. }
  61. testrun() {
  62. configcompression '.' 'xz'
  63. msgmsg "Testcase: setup the base with: $*"
  64. find aptarchive -name 'Packages*' -type f -delete
  65. cp "${PKGFILE}" aptarchive/Packages
  66. compressfile 'aptarchive/Packages'
  67. generatereleasefiles
  68. signreleasefiles
  69. rm -rf aptarchive/Packages.diff rootdir/var/lib/apt/lists rootdir/var/lib/apt/lists-bak
  70. testsuccess aptget update "$@"
  71. cp -a rootdir/var/lib/apt/lists rootdir/var/lib/apt/lists-bak
  72. testnopackage newstuff
  73. testsuccessequal "$(cat "${PKGFILE}")
  74. " aptcache show apt oldstuff
  75. configcompression '.' 'gz'
  76. # see if the code deals properly with leftover partial files
  77. partialleftovers
  78. msgmsg "Testcase: apply with one patch: $*"
  79. find aptarchive -name 'Packages*' -type f -delete
  80. cp "${PKGFILE}-new" aptarchive/Packages
  81. compressfile 'aptarchive/Packages'
  82. mkdir -p aptarchive/Packages.diff
  83. PATCHFILE="aptarchive/Packages.diff/$(date +%Y-%m-%d-%H%M.%S)"
  84. diff -e "${PKGFILE}" "${PKGFILE}-new" > "${PATCHFILE}" || true
  85. cat "$PATCHFILE" | gzip > "${PATCHFILE}.gz"
  86. PATCHINDEX='aptarchive/Packages.diff/Index'
  87. echo "SHA256-Current: $(sha256sum "${PKGFILE}-new" | cut -d' ' -f 1) $(stat -c%s "${PKGFILE}-new")
  88. SHA256-History:
  89. 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b 33053002 2010-08-18-2013.28
  90. $(sha256sum "$PKGFILE" | cut -d' ' -f 1) $(stat -c%s "$PKGFILE") $(basename "$PATCHFILE")
  91. SHA256-Patches:
  92. e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 19722 2010-08-18-2013.28
  93. $(sha256sum "$PATCHFILE" | cut -d' ' -f 1) $(stat -c%s "$PATCHFILE") $(basename "$PATCHFILE")
  94. SHA256-Download:
  95. d2a1b33187ed2d248eeae3b1223ea71791ea35f2138a713ed371332a6421f467 197 2010-08-18-2013.28.gz
  96. $(sha256sum "${PATCHFILE}.gz" | cut -d' ' -f 1) $(stat -c%s "${PATCHFILE}.gz") $(basename "${PATCHFILE}.gz")" > "$PATCHINDEX"
  97. generatereleasefiles '+1hour'
  98. signreleasefiles
  99. find aptarchive -name 'Packages*' -type f -delete
  100. wasmergeused "$@"
  101. testnopackage oldstuff
  102. testsuccessequal "$(cat "${PKGFILE}-new")
  103. " aptcache show apt newstuff
  104. msgmsg "Testcase: apply with compressed Index and one patch: $*"
  105. cp "${PKGFILE}-new" aptarchive/Packages
  106. compressfile 'aptarchive/Packages'
  107. cat "$PATCHINDEX" | gzip > "${PATCHINDEX}.gz"
  108. generatereleasefiles '+1hour'
  109. signreleasefiles
  110. find aptarchive -name 'Packages*' -type f -delete
  111. rm "$PATCHINDEX"
  112. rm -rf rootdir/var/lib/apt/lists
  113. cp -a rootdir/var/lib/apt/lists-bak rootdir/var/lib/apt/lists
  114. wasmergeused "$@"
  115. testnopackage oldstuff
  116. testsuccessequal "$(cat "${PKGFILE}-new")
  117. " aptcache show apt newstuff
  118. msgmsg "Testcase: apply with by-hash compressed Index and one patch: $*"
  119. local NORMAL="$(readlink -f "./aptarchive/Packages.diff")"
  120. local BYHASH="${NORMAL}/by-hash/SHA256"
  121. mkdir -p "${BYHASH}"
  122. find "${NORMAL}/" -maxdepth 1 -name "Index*" -exec mv '{}' "$BYHASH" \;
  123. ln -s "${BYHASH}/Index.gz" "${BYHASH}/$(sha256sum "${BYHASH}/Index.gz" | cut -f1 -d' ')"
  124. echo 'foobar' > "${BYHASH}/$(sha256sum "$PATCHFILE" | cut -f1 -d' ')"
  125. echo 'foobar' > "${BYHASH}/$(sha256sum "${PATCHFILE}.gz" | cut -f1 -d' ')"
  126. rm -rf rootdir/var/lib/apt/lists
  127. cp -a rootdir/var/lib/apt/lists-bak rootdir/var/lib/apt/lists
  128. wasmergeused "$@" -o Acquire::By-Hash=force
  129. testnopackage oldstuff
  130. testsuccessequal "$(cat "${PKGFILE}-new")
  131. " aptcache show apt newstuff
  132. msgmsg "Testcase: SHA1-only patches are not used: $*"
  133. find aptarchive -name 'Packages*' -type f -delete
  134. cp "${PKGFILE}-new" aptarchive/Packages
  135. compressfile 'aptarchive/Packages'
  136. mkdir -p aptarchive/Packages.diff
  137. PATCHFILE="aptarchive/Packages.diff/$(date +%Y-%m-%d-%H%M.%S)"
  138. diff -e "${PKGFILE}" "${PKGFILE}-new" > "${PATCHFILE}" || true
  139. cat "$PATCHFILE" | gzip > "${PATCHFILE}.gz"
  140. PATCHINDEX='aptarchive/Packages.diff/Index'
  141. echo "SHA1-Current: $(sha1sum "${PKGFILE}-new" | cut -d' ' -f 1) $(stat -c%s "${PKGFILE}-new")
  142. SHA1-History:
  143. 9f4148e06d7faa37062994ff10d0c842d7017513 33053002 2010-08-18-2013.28
  144. $(sha1sum "$PKGFILE" | cut -d' ' -f 1) $(stat -c%s "$PKGFILE") $(basename "$PATCHFILE")
  145. SHA1-Patches:
  146. 7651fc0ac57cd83d41c63195a9342e2db5650257 19722 2010-08-18-2013.28
  147. $(sha1sum "$PATCHFILE" | cut -d' ' -f 1) $(stat -c%s "$PATCHFILE") $(basename "$PATCHFILE")
  148. SHA1-Download:
  149. 2365ac0ac57cde3d43c63145e8251a3bd5410213 197 2010-08-18-2013.28.gz
  150. $(sha1sum "${PATCHFILE}.gz" | cut -d' ' -f 1) $(stat -c%s "${PATCHFILE}.gz") $(basename "${PATCHFILE}.gz")" > "$PATCHINDEX"
  151. generatereleasefiles '+1hour'
  152. signreleasefiles
  153. rm -rf rootdir/var/lib/apt/lists
  154. cp -a rootdir/var/lib/apt/lists-bak rootdir/var/lib/apt/lists
  155. wasmergeused "$@" -o test::cannot-use-pdiff=1
  156. testnopackage oldstuff
  157. testsuccessequal "$(cat "${PKGFILE}-new")
  158. " aptcache show apt newstuff
  159. msgmsg "Testcase: no download-hashes patches are not used: $*"
  160. find aptarchive -name 'Packages*' -type f -delete
  161. cp "${PKGFILE}-new" aptarchive/Packages
  162. compressfile 'aptarchive/Packages'
  163. mkdir -p aptarchive/Packages.diff
  164. PATCHFILE="aptarchive/Packages.diff/$(date +%Y-%m-%d-%H%M.%S)"
  165. diff -e "${PKGFILE}" "${PKGFILE}-new" > "${PATCHFILE}" || true
  166. cat "$PATCHFILE" | gzip > "${PATCHFILE}.gz"
  167. PATCHINDEX='aptarchive/Packages.diff/Index'
  168. echo "SHA256-Current: $(sha256sum "${PKGFILE}-new" | cut -d' ' -f 1) $(stat -c%s "${PKGFILE}-new")
  169. SHA256-History:
  170. 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b 33053002 2010-08-18-2013.28
  171. $(sha256sum "$PKGFILE" | cut -d' ' -f 1) $(stat -c%s "$PKGFILE") $(basename "$PATCHFILE")
  172. SHA256-Patches:
  173. e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 19722 2010-08-18-2013.28
  174. $(sha256sum "$PATCHFILE" | cut -d' ' -f 1) $(stat -c%s "$PATCHFILE") $(basename "$PATCHFILE")" > "$PATCHINDEX"
  175. generatereleasefiles '+1hour'
  176. signreleasefiles
  177. rm -rf rootdir/var/lib/apt/lists
  178. cp -a rootdir/var/lib/apt/lists-bak rootdir/var/lib/apt/lists
  179. wasmergeused "$@" -o test::cannot-use-pdiff=1
  180. testnopackage oldstuff
  181. testsuccessequal "$(cat "${PKGFILE}-new")
  182. " aptcache show apt newstuff
  183. msgmsg "Testcase: apply with two patches: $*"
  184. cp "${PKGFILE}-new" aptarchive/Packages
  185. echo '
  186. Package: futurestuff
  187. Version: 1.0
  188. Architecture: i386
  189. Maintainer: Joe Sixpack <joe@example.org>
  190. Installed-Size: 202
  191. Filename: pool/futurestuff_1.0_i386.deb
  192. Size: 202200
  193. SHA256: b46fd154615edaae5ba33c56a5cc0e7deaef23e2da3e4f129727fd660f28f050
  194. Description: some cool and shiny future stuff
  195. This package will appear in the next^2 mirror update
  196. Description-md5: d5f89fbbc2ce34c455dfee9b67d82b6b' >> aptarchive/Packages
  197. compressfile 'aptarchive/Packages'
  198. PATCHFILE2="aptarchive/Packages.diff/$(date -d 'now + 1hour' '+%Y-%m-%d-%H%M.%S')"
  199. diff -e "${PKGFILE}-new" aptarchive/Packages > "${PATCHFILE2}" || true
  200. cat "$PATCHFILE2" | gzip > "${PATCHFILE2}.gz"
  201. echo "SHA256-Current: $(sha256sum aptarchive/Packages | cut -d' ' -f 1) $(stat -c%s aptarchive/Packages)
  202. SHA256-History:
  203. 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b 33053002 2010-08-18-2013.28
  204. $(sha256sum "$PKGFILE" | cut -d' ' -f 1) $(stat -c%s "$PKGFILE") $(basename "$PATCHFILE")
  205. $(sha256sum "${PKGFILE}-new" | cut -d' ' -f 1) $(stat -c%s "${PKGFILE}-new") $(basename "${PATCHFILE2}")
  206. SHA256-Patches:
  207. e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 19722 2010-08-18-2013.28
  208. $(sha256sum "$PATCHFILE" | cut -d' ' -f 1) $(stat -c%s "$PATCHFILE") $(basename "$PATCHFILE")
  209. $(sha256sum "${PATCHFILE2}" | cut -d' ' -f 1) $(stat -c%s "${PATCHFILE2}") $(basename "${PATCHFILE2}")
  210. SHA256-Download:
  211. d2a1b33187ed2d248eeae3b1223ea71791ea35f2138a713ed371332a6421f467 197 2010-08-18-2013.28.gz
  212. $(sha256sum "${PATCHFILE}.gz" | cut -d' ' -f 1) $(stat -c%s "${PATCHFILE}.gz") $(basename "${PATCHFILE}.gz")
  213. $(sha256sum "${PATCHFILE2}.gz" | cut -d' ' -f 1) $(stat -c%s "${PATCHFILE2}.gz") $(basename "${PATCHFILE2}.gz")" > "$PATCHINDEX"
  214. generatereleasefiles '+2hour'
  215. signreleasefiles
  216. cp -a aptarchive/Packages Packages-future
  217. find aptarchive -name 'Packages*' -type f -delete
  218. rm -rf rootdir/var/lib/apt/lists
  219. cp -a rootdir/var/lib/apt/lists-bak rootdir/var/lib/apt/lists
  220. wasmergeused "$@"
  221. testnopackage oldstuff
  222. testsuccessequal "$(cat Packages-future)
  223. " aptcache show apt newstuff futurestuff
  224. # we reuse the archive state of the previous test here
  225. msgmsg "Testcase: pdiff handling is stopped if transaction fails $*"
  226. rm -rf rootdir/var/lib/apt/lists
  227. cp -a rootdir/var/lib/apt/lists-bak rootdir/var/lib/apt/lists
  228. cp Packages-future aptarchive/Packages
  229. rm -f rootdir/var/lib/apt/lists/*_Contents-*
  230. webserverconfig 'aptwebserver::overwrite::.*Contents-.*::filename' '/hacked-i386.gz'
  231. # This should work in at least 4% of the cases...
  232. for i in $(seq 25); do
  233. testfailure apt update "$@"
  234. if ! grep 'rred:600' rootdir/tmp/testfailure.output; then
  235. break
  236. fi
  237. done
  238. webserverconfig 'aptwebserver::overwrite::.*Contents-.*::filename' '/Contents-i386.gz'
  239. cp rootdir/tmp/testfailure.output patchdownload.output
  240. testfailure grep 'rred:600' patchdownload.output
  241. testnopackage newstuff futurestuff
  242. testsuccessequal "$(cat "${PKGFILE}")
  243. " aptcache show apt oldstuff
  244. # we reuse the entire state of the previous test here
  245. msgmsg "Testcase: good files from previous fails are picked up from partial: $*"
  246. wasmergeused "$@"
  247. testfailure grep '^GET /Packages.diff/Index HTTP/1.1' rootdir/tmp/testsuccess.output
  248. testnopackage oldstuff
  249. testsuccessequal "$(cat Packages-future)
  250. " aptcache show apt newstuff futurestuff
  251. # we reuse the archive state of the previous test here
  252. msgmsg "Testcase: downloading a patch fails, but successful fallback: $*"
  253. rm -rf rootdir/var/lib/apt/lists
  254. cp -a rootdir/var/lib/apt/lists-bak rootdir/var/lib/apt/lists
  255. cp Packages-future aptarchive/Packages
  256. rm "${PATCHFILE}.gz"
  257. testsuccess apt update "$@"
  258. cp rootdir/tmp/testsuccess.output patchdownload.output
  259. testsuccess grep '^Falling back to normal index file acquire' patchdownload.output
  260. testnopackage oldstuff
  261. testsuccessequal "$(cat Packages-future)
  262. " aptcache show apt newstuff futurestuff
  263. msgmsg "Testcase: patch applying fails, but successful fallback: $*"
  264. rm -rf rootdir/var/lib/apt/lists
  265. cp -a rootdir/var/lib/apt/lists-bak rootdir/var/lib/apt/lists
  266. cp "${PKGFILE}-new" aptarchive/Packages
  267. compressfile 'aptarchive/Packages'
  268. mkdir -p aptarchive/Packages.diff
  269. PATCHFILE="aptarchive/Packages.diff/$(date +%Y-%m-%d-%H%M.%S)"
  270. diff -e "${PKGFILE}" "${PKGFILE}-new" > "${PATCHFILE}" || true
  271. cat "$PATCHFILE" | gzip > "${PATCHFILE}.gz"
  272. PATCHINDEX='aptarchive/Packages.diff/Index'
  273. echo "SHA256-Current: $(sha256sum "${PKGFILE}-new" | cut -d' ' -f 1) $(stat -c%s "${PKGFILE}-new")
  274. SHA256-History:
  275. 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b 33053002 2010-08-18-2013.28
  276. $(sha256sum "$PKGFILE" | cut -d' ' -f 1) $(stat -c%s "$PKGFILE") $(basename "$PATCHFILE")
  277. SHA256-Patches:
  278. e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 19722 2010-08-18-2013.28
  279. $(sha256sum "$PATCHFILE" | cut -d' ' -f 1) $(stat -c%s "$PATCHFILE") $(basename "$PATCHFILE")
  280. SHA256-Download:
  281. d2a1b33187ed2d248eeae3b1223ea71791ea35f2138a713ed371332a6421f467 197 2010-08-18-2013.28.gz
  282. $(sha256sum "${PATCHFILE}.gz" | cut -d' ' -f 1) $(stat -c%s "${PATCHFILE}.gz") $(basename "${PATCHFILE}.gz")" > "$PATCHINDEX"
  283. breakfiles "$PATCHFILE" "${PATCHFILE}.gz"
  284. generatereleasefiles '+1hour'
  285. signreleasefiles
  286. testsuccess apt update "$@"
  287. cp -f rootdir/tmp/testsuccess.output rootdir/tmp/aptupdate.output
  288. testsuccess grep 'Hash Sum mismatch' rootdir/tmp/aptupdate.output
  289. testnopackage oldstuff
  290. testsuccessequal "$(cat "${PKGFILE}-new")
  291. " aptcache show apt newstuff
  292. msgmsg "Testcase: pdiff patch bigger than index itself: $*"
  293. rm -rf rootdir/var/lib/apt/lists
  294. cp -a rootdir/var/lib/apt/lists-bak rootdir/var/lib/apt/lists
  295. # the general testcode checks for leftovers in partial, but as this file
  296. # is never touched in this test as it exits earlier its not a leftover…
  297. rm -f rootdir/var/lib/apt/lists/partial/localhost:${APTHTTPPORT}_Packages
  298. cp "${PKGFILE}-new" aptarchive/Packages
  299. compressfile 'aptarchive/Packages'
  300. mkdir -p aptarchive/Packages.diff
  301. PATCHFILE="aptarchive/Packages.diff/$(date +%Y-%m-%d-%H%M.%S)"
  302. diff -e "${PKGFILE}" "${PKGFILE}-new" > "${PATCHFILE}" || true
  303. cat "$PATCHFILE" | gzip > "${PATCHFILE}.gz"
  304. PATCHINDEX='aptarchive/Packages.diff/Index'
  305. echo "SHA256-Current: $(sha256sum "${PKGFILE}-new" | cut -d' ' -f 1) $(stat -c%s "${PKGFILE}-new")
  306. SHA256-History:
  307. 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b 33053002 2010-08-18-2013.28
  308. $(sha256sum "$PKGFILE" | cut -d' ' -f 1) $(stat -c%s "$PKGFILE") $(basename "$PATCHFILE")
  309. SHA256-Patches:
  310. e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 19722 2010-08-18-2013.28
  311. $(sha256sum "$PATCHFILE" | cut -d' ' -f 1) $(stat -c%s "$PATCHFILE") $(basename "$PATCHFILE")
  312. SHA256-Download:
  313. d2a1b33187ed2d248eeae3b1223ea71791ea35f2138a713ed371332a6421f467 197 2010-08-18-2013.28.gz
  314. $(sha256sum "${PATCHFILE}.gz" | cut -d' ' -f 1) $(stat -c%s "${PATCHFILE}.gz")000 $(basename "${PATCHFILE}.gz")" > "$PATCHINDEX"
  315. generatereleasefiles '+1hour'
  316. signreleasefiles
  317. wasmergeused "$@" -o test::cannot-use-pdiff=1
  318. testsuccess grep 'bytes, but limit is' rootdir/tmp/aptupdate.output
  319. testnopackage oldstuff
  320. testsuccessequal "$(cat "${PKGFILE}-new")
  321. " aptcache show apt newstuff
  322. msgmsg "Testcase: pdiff index failed to download: $*"
  323. rm -rf rootdir/var/lib/apt/lists
  324. cp -a rootdir/var/lib/apt/lists-bak rootdir/var/lib/apt/lists
  325. cp "${PKGFILE}-new" aptarchive/Packages
  326. compressfile 'aptarchive/Packages'
  327. mkdir -p aptarchive/Packages.diff
  328. PATCHFILE="aptarchive/Packages.diff/$(date +%Y-%m-%d-%H%M.%S)"
  329. diff -e "${PKGFILE}" "${PKGFILE}-new" > "${PATCHFILE}" || true
  330. cat "$PATCHFILE" | gzip > "${PATCHFILE}.gz"
  331. PATCHINDEX='aptarchive/Packages.diff/Index'
  332. echo "SHA256-Current: $(sha256sum "${PKGFILE}-new" | cut -d' ' -f 1) $(stat -c%s "${PKGFILE}-new")
  333. SHA256-History:
  334. 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b 33053002 2010-08-18-2013.28
  335. $(sha256sum "$PKGFILE" | cut -d' ' -f 1) $(stat -c%s "$PKGFILE") $(basename "$PATCHFILE")
  336. SHA256-Patches:
  337. e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 19722 2010-08-18-2013.28
  338. $(sha256sum "$PATCHFILE" | cut -d' ' -f 1) $(stat -c%s "$PATCHFILE") $(basename "$PATCHFILE")
  339. SHA256-Download:
  340. d2a1b33187ed2d248eeae3b1223ea71791ea35f2138a713ed371332a6421f467 197 2010-08-18-2013.28.gz
  341. $(sha256sum "${PATCHFILE}.gz" | cut -d' ' -f 1) $(stat -c%s "${PATCHFILE}.gz") $(basename "${PATCHFILE}.gz")" > "$PATCHINDEX"
  342. generatereleasefiles '+1hour'
  343. signreleasefiles
  344. # we let it fail by removing the files so the webserver reports 404
  345. rm -f "$PATCHINDEX" "$PATCHFILE" "${PATCHFILE}.gz"
  346. wasmergeused "$@" -o test::cannot-use-pdiff=1
  347. testsuccess grep '400%20URI%20Failure.*FailReason.*HttpError404.*Packages\.diff/Index' rootdir/tmp/aptupdate.output
  348. testnopackage oldstuff
  349. testsuccessequal "$(cat "${PKGFILE}-new")
  350. " aptcache show apt newstuff
  351. }
  352. echo 'Debug::pkgAcquire::Diffs "true";
  353. Debug::Acquire::Transaction "true";
  354. Debug::pkgAcquire::Worker "true";
  355. Debug::Acquire::http "true";
  356. Debug::pkgAcquire "true";
  357. Debug::Acquire::rred "true";' > rootdir/etc/apt/apt.conf.d/rreddebug.conf
  358. testcase() {
  359. testrun -o Acquire::PDiffs::Merge=0 -o APT::Get::List-Cleanup=1 "$@"
  360. testrun -o Acquire::PDiffs::Merge=1 -o APT::Get::List-Cleanup=1 "$@"
  361. testrun -o Acquire::PDiffs::Merge=0 -o APT::Get::List-Cleanup=0 "$@"
  362. testrun -o Acquire::PDiffs::Merge=1 -o APT::Get::List-Cleanup=0 "$@"
  363. }
  364. generatepartialleftovers() {
  365. for f in "$@"; do
  366. cat "${PKGFILE}" "${PKGFILE}" > "rootdir/var/lib/apt/lists-bak/partial/localhost:${APTHTTPPORT}_${f}"
  367. printf '\n\nInvalid\nStanza: yes\n\n' >> "rootdir/var/lib/apt/lists-bak/partial/localhost:${APTHTTPPORT}_${f}"
  368. done
  369. }
  370. partialleftovers() { generatepartialleftovers 'Packages' 'Packages-patched'; }
  371. aptautotest_apt_update() { aptautotest_aptget_update "$@"; testsuccess test -e "rootdir/var/lib/apt/lists/localhost:${APTHTTPPORT}_Packages"; }
  372. testcase -o Acquire::IndexTargets::deb::Packages::KeepCompressed=false
  373. partialleftovers() { generatepartialleftovers "Packages.${LOWCOSTEXT}" "Packages-patched.${LOWCOSTEXT}"; }
  374. aptautotest_apt_update() { aptautotest_aptget_update "$@"; testsuccess test -e "rootdir/var/lib/apt/lists/localhost:${APTHTTPPORT}_Packages.$LOWCOSTEXT"; }
  375. testcase -o Acquire::IndexTargets::deb::Packages::KeepCompressed=true
  376. partialleftovers() { generatepartialleftovers "redirectme_Packages.${LOWCOSTEXT}" "redirectme_Packages-patched.${LOWCOSTEXT}"; }
  377. # redirect the InRelease file only – the other files are auto-redirected by apt
  378. webserverconfig 'aptwebserver::redirect::replace::/redirectme/I' "http://0.0.0.0:${APTHTTPPORT}/I"
  379. rewritesourceslist "http://localhost:${APTHTTPPORT}/redirectme"
  380. aptautotest_apt_update() {
  381. aptautotest_aptget_update "$@"
  382. testsuccess test -e "rootdir/var/lib/apt/lists/localhost:${APTHTTPPORT}_redirectme_Packages.$LOWCOSTEXT"
  383. testempty find rootdir/var/lib/apt/lists -type f \! \( -name lock -o -name '*_redirectme_*' \)
  384. }
  385. testcase -o Acquire::IndexTargets::deb::Packages::KeepCompressed=true