You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

rred.cc 20 KiB

implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
implement POC client-side merging of pdiffs via apt-file The idea of pdiffs is to avoid downloading the hole file by patching the existing index. This works very well, but becomes slow if a lot of patches needs to be applied to reconstruct an up-to-date index and in recent years more and more dinstall (or similar) runs are executed creating more and more pdiffs in the same amount of time, so pdiffs became less useful. The solution is simple: Reduce the amount of patches (which are very small) which need to be applied on top of the index we have available (which is usually pretty big). This can be done in two ways: Either merge the patches on the server-side so that the client has to download only one patch or the patches are all downloaded and merged on the client-side. The first needs a client who is doing one step at a time who can also skip patches if it needs (APT supports this for a long time now). The later is implemented by this commit, but depends on the server NOT merging the patches and the patches being in a strict order in which no patch is skipped. This is traditionally the case for dak, but other repository creators support merging – e.g. reprepro (which helpfully adds a flag indicating that the patches are merged). To support both or even mixes a client needs more information which isn't available for now. This POC uses the external diffindex-rred included in apt-file to do the heavy lifting of merging & applying all patches in one pass, hence to test this feature apt-file needs to be installed.
7 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788
  1. // Copyright (c) 2014 Anthony Towns
  2. //
  3. // This program is free software; you can redistribute it and/or modify
  4. // it under the terms of the GNU General Public License as published by
  5. // the Free Software Foundation; either version 2 of the License, or
  6. // (at your option) any later version.
  7. #include <config.h>
  8. #include "aptmethod.h"
  9. #include <apt-pkg/configuration.h>
  10. #include <apt-pkg/error.h>
  11. #include <apt-pkg/fileutl.h>
  12. #include <apt-pkg/hashes.h>
  13. #include <apt-pkg/init.h>
  14. #include <apt-pkg/strutl.h>
  15. #include <iostream>
  16. #include <list>
  17. #include <string>
  18. #include <vector>
  19. #include <stddef.h>
  20. #include <assert.h>
  21. #include <errno.h>
  22. #include <stdio.h>
  23. #include <stdlib.h>
  24. #include <string.h>
  25. #include <sys/stat.h>
  26. #include <sys/time.h>
  27. #include <apti18n.h>
  28. #define BLOCK_SIZE (512*1024)
  29. class MemBlock {
  30. char *start;
  31. size_t size;
  32. char *free;
  33. MemBlock *next;
  34. explicit MemBlock(size_t size) : size(size), next(NULL)
  35. {
  36. free = start = new char[size];
  37. }
  38. size_t avail(void) { return size - (free - start); }
  39. public:
  40. MemBlock(void) {
  41. free = start = new char[BLOCK_SIZE];
  42. size = BLOCK_SIZE;
  43. next = NULL;
  44. }
  45. ~MemBlock() {
  46. delete [] start;
  47. delete next;
  48. }
  49. void clear(void) {
  50. free = start;
  51. if (next)
  52. next->clear();
  53. }
  54. char *add_easy(char *src, size_t len, char *last)
  55. {
  56. if (last) {
  57. for (MemBlock *k = this; k; k = k->next) {
  58. if (k->free == last) {
  59. if (len <= k->avail()) {
  60. char *n = k->add(src, len);
  61. assert(last == n);
  62. if (last == n)
  63. return NULL;
  64. return n;
  65. } else {
  66. break;
  67. }
  68. } else if (last >= start && last < free) {
  69. break;
  70. }
  71. }
  72. }
  73. return add(src, len);
  74. }
  75. char *add(char *src, size_t len) {
  76. if (len > avail()) {
  77. if (!next) {
  78. if (len > BLOCK_SIZE) {
  79. next = new MemBlock(len);
  80. } else {
  81. next = new MemBlock;
  82. }
  83. }
  84. return next->add(src, len);
  85. }
  86. char *dst = free;
  87. free += len;
  88. memcpy(dst, src, len);
  89. return dst;
  90. }
  91. };
  92. struct Change {
  93. /* Ordering:
  94. *
  95. * 1. write out <offset> lines unchanged
  96. * 2. skip <del_cnt> lines from source
  97. * 3. write out <add_cnt> lines (<add>/<add_len>)
  98. */
  99. size_t offset;
  100. size_t del_cnt;
  101. size_t add_cnt; /* lines */
  102. size_t add_len; /* bytes */
  103. char *add;
  104. explicit Change(size_t off)
  105. {
  106. offset = off;
  107. del_cnt = add_cnt = add_len = 0;
  108. add = NULL;
  109. }
  110. /* actually, don't write <lines> lines from <add> */
  111. void skip_lines(size_t lines)
  112. {
  113. while (lines > 0) {
  114. char *s = (char*) memchr(add, '\n', add_len);
  115. assert(s != NULL);
  116. s++;
  117. add_len -= (s - add);
  118. add_cnt--;
  119. lines--;
  120. if (add_len == 0) {
  121. add = NULL;
  122. assert(add_cnt == 0);
  123. assert(lines == 0);
  124. } else {
  125. add = s;
  126. assert(add_cnt > 0);
  127. }
  128. }
  129. }
  130. };
  131. class FileChanges {
  132. std::list<struct Change> changes;
  133. std::list<struct Change>::iterator where;
  134. size_t pos; // line number is as far left of iterator as possible
  135. bool pos_is_okay(void) const
  136. {
  137. #ifdef POSDEBUG
  138. size_t cpos = 0;
  139. std::list<struct Change>::const_iterator x;
  140. for (x = changes.begin(); x != where; ++x) {
  141. assert(x != changes.end());
  142. cpos += x->offset + x->add_cnt;
  143. }
  144. return cpos == pos;
  145. #else
  146. return true;
  147. #endif
  148. }
  149. public:
  150. FileChanges() {
  151. where = changes.end();
  152. pos = 0;
  153. }
  154. std::list<struct Change>::iterator begin(void) { return changes.begin(); }
  155. std::list<struct Change>::iterator end(void) { return changes.end(); }
  156. std::list<struct Change>::reverse_iterator rbegin(void) { return changes.rbegin(); }
  157. std::list<struct Change>::reverse_iterator rend(void) { return changes.rend(); }
  158. void add_change(Change c) {
  159. assert(pos_is_okay());
  160. go_to_change_for(c.offset);
  161. assert(pos + where->offset == c.offset);
  162. if (c.del_cnt > 0)
  163. delete_lines(c.del_cnt);
  164. assert(pos + where->offset == c.offset);
  165. if (c.add_len > 0) {
  166. assert(pos_is_okay());
  167. if (where->add_len > 0)
  168. new_change();
  169. assert(where->add_len == 0 && where->add_cnt == 0);
  170. where->add_len = c.add_len;
  171. where->add_cnt = c.add_cnt;
  172. where->add = c.add;
  173. }
  174. assert(pos_is_okay());
  175. merge();
  176. assert(pos_is_okay());
  177. }
  178. private:
  179. void merge(void)
  180. {
  181. while (where->offset == 0 && where != changes.begin()) {
  182. left();
  183. }
  184. std::list<struct Change>::iterator next = where;
  185. ++next;
  186. while (next != changes.end() && next->offset == 0) {
  187. where->del_cnt += next->del_cnt;
  188. next->del_cnt = 0;
  189. if (next->add == NULL) {
  190. next = changes.erase(next);
  191. } else if (where->add == NULL) {
  192. where->add = next->add;
  193. where->add_len = next->add_len;
  194. where->add_cnt = next->add_cnt;
  195. next = changes.erase(next);
  196. } else {
  197. ++next;
  198. }
  199. }
  200. }
  201. void go_to_change_for(size_t line)
  202. {
  203. while(where != changes.end()) {
  204. if (line < pos) {
  205. left();
  206. continue;
  207. }
  208. if (pos + where->offset + where->add_cnt <= line) {
  209. right();
  210. continue;
  211. }
  212. // line is somewhere in this slot
  213. if (line < pos + where->offset) {
  214. break;
  215. } else if (line == pos + where->offset) {
  216. return;
  217. } else {
  218. split(line - pos);
  219. right();
  220. return;
  221. }
  222. }
  223. /* it goes before this patch */
  224. insert(line-pos);
  225. }
  226. void new_change(void) { insert(where->offset); }
  227. void insert(size_t offset)
  228. {
  229. assert(pos_is_okay());
  230. assert(where == changes.end() || offset <= where->offset);
  231. if (where != changes.end())
  232. where->offset -= offset;
  233. changes.insert(where, Change(offset));
  234. --where;
  235. assert(pos_is_okay());
  236. }
  237. void split(size_t offset)
  238. {
  239. assert(pos_is_okay());
  240. assert(where->offset < offset);
  241. assert(offset < where->offset + where->add_cnt);
  242. size_t keep_lines = offset - where->offset;
  243. Change before(*where);
  244. where->del_cnt = 0;
  245. where->offset = 0;
  246. where->skip_lines(keep_lines);
  247. before.add_cnt = keep_lines;
  248. before.add_len -= where->add_len;
  249. changes.insert(where, before);
  250. --where;
  251. assert(pos_is_okay());
  252. }
  253. void delete_lines(size_t cnt)
  254. {
  255. std::list<struct Change>::iterator x = where;
  256. assert(pos_is_okay());
  257. while (cnt > 0)
  258. {
  259. size_t del;
  260. del = x->add_cnt;
  261. if (del > cnt)
  262. del = cnt;
  263. x->skip_lines(del);
  264. cnt -= del;
  265. ++x;
  266. if (x == changes.end()) {
  267. del = cnt;
  268. } else {
  269. del = x->offset;
  270. if (del > cnt)
  271. del = cnt;
  272. x->offset -= del;
  273. }
  274. where->del_cnt += del;
  275. cnt -= del;
  276. }
  277. assert(pos_is_okay());
  278. }
  279. void left(void) {
  280. assert(pos_is_okay());
  281. --where;
  282. pos -= where->offset + where->add_cnt;
  283. assert(pos_is_okay());
  284. }
  285. void right(void) {
  286. assert(pos_is_okay());
  287. pos += where->offset + where->add_cnt;
  288. ++where;
  289. assert(pos_is_okay());
  290. }
  291. };
  292. class Patch {
  293. FileChanges filechanges;
  294. MemBlock add_text;
  295. static bool retry_fwrite(char *b, size_t l, FileFd &f, Hashes * const start_hash, Hashes * const end_hash = nullptr) APT_NONNULL(1)
  296. {
  297. if (f.Write(b, l) == false)
  298. return false;
  299. if (start_hash)
  300. start_hash->Add((unsigned char*)b, l);
  301. if (end_hash)
  302. end_hash->Add((unsigned char*)b, l);
  303. return true;
  304. }
  305. static void dump_rest(FileFd &o, FileFd &i,
  306. Hashes * const start_hash, Hashes * const end_hash)
  307. {
  308. char buffer[BLOCK_SIZE];
  309. unsigned long long l = 0;
  310. while (i.Read(buffer, sizeof(buffer), &l)) {
  311. if (l ==0 || !retry_fwrite(buffer, l, o, start_hash, end_hash))
  312. break;
  313. }
  314. }
  315. static void dump_lines(FileFd &o, FileFd &i, size_t n,
  316. Hashes * const start_hash, Hashes * const end_hash)
  317. {
  318. char buffer[BLOCK_SIZE];
  319. while (n > 0) {
  320. if (i.ReadLine(buffer, sizeof(buffer)) == NULL)
  321. buffer[0] = '\0';
  322. size_t const l = strlen(buffer);
  323. if (l == 0 || buffer[l-1] == '\n')
  324. n--;
  325. retry_fwrite(buffer, l, o, start_hash, end_hash);
  326. }
  327. }
  328. static void skip_lines(FileFd &i, int n, Hashes * const start_hash)
  329. {
  330. char buffer[BLOCK_SIZE];
  331. while (n > 0) {
  332. if (i.ReadLine(buffer, sizeof(buffer)) == NULL)
  333. buffer[0] = '\0';
  334. size_t const l = strlen(buffer);
  335. if (l == 0 || buffer[l-1] == '\n')
  336. n--;
  337. if (start_hash)
  338. start_hash->Add((unsigned char*)buffer, l);
  339. }
  340. }
  341. static void dump_mem(FileFd &o, char *p, size_t s, Hashes *hash) APT_NONNULL(2) {
  342. retry_fwrite(p, s, o, nullptr, hash);
  343. }
  344. public:
  345. bool read_diff(FileFd &f, Hashes * const h)
  346. {
  347. char buffer[BLOCK_SIZE];
  348. bool cmdwanted = true;
  349. Change ch(std::numeric_limits<size_t>::max());
  350. if (f.ReadLine(buffer, sizeof(buffer)) == NULL)
  351. return _error->Error("Reading first line of patchfile %s failed", f.Name().c_str());
  352. do {
  353. if (h != NULL)
  354. h->Add(buffer);
  355. if (cmdwanted) {
  356. char *m, *c;
  357. size_t s, e;
  358. errno = 0;
  359. s = strtoul(buffer, &m, 10);
  360. if (unlikely(m == buffer || s == std::numeric_limits<unsigned long>::max() || errno != 0))
  361. return _error->Error("Parsing patchfile %s failed: Expected an effected line start", f.Name().c_str());
  362. else if (*m == ',') {
  363. ++m;
  364. e = strtol(m, &c, 10);
  365. if (unlikely(m == c || e == std::numeric_limits<unsigned long>::max() || errno != 0))
  366. return _error->Error("Parsing patchfile %s failed: Expected an effected line end", f.Name().c_str());
  367. if (unlikely(e < s))
  368. return _error->Error("Parsing patchfile %s failed: Effected lines end %lu is before start %lu", f.Name().c_str(), e, s);
  369. } else {
  370. e = s;
  371. c = m;
  372. }
  373. if (s > ch.offset)
  374. return _error->Error("Parsing patchfile %s failed: Effected line is after previous effected line", f.Name().c_str());
  375. switch(*c) {
  376. case 'a':
  377. cmdwanted = false;
  378. ch.add = NULL;
  379. ch.add_cnt = 0;
  380. ch.add_len = 0;
  381. ch.offset = s;
  382. ch.del_cnt = 0;
  383. break;
  384. case 'c':
  385. if (unlikely(s == 0))
  386. return _error->Error("Parsing patchfile %s failed: Change command can't effect line zero", f.Name().c_str());
  387. cmdwanted = false;
  388. ch.add = NULL;
  389. ch.add_cnt = 0;
  390. ch.add_len = 0;
  391. ch.offset = s - 1;
  392. ch.del_cnt = e - s + 1;
  393. break;
  394. case 'd':
  395. if (unlikely(s == 0))
  396. return _error->Error("Parsing patchfile %s failed: Delete command can't effect line zero", f.Name().c_str());
  397. ch.offset = s - 1;
  398. ch.del_cnt = e - s + 1;
  399. ch.add = NULL;
  400. ch.add_cnt = 0;
  401. ch.add_len = 0;
  402. filechanges.add_change(ch);
  403. break;
  404. default:
  405. return _error->Error("Parsing patchfile %s failed: Unknown command", f.Name().c_str());
  406. }
  407. } else { /* !cmdwanted */
  408. if (strcmp(buffer, ".\n") == 0) {
  409. cmdwanted = true;
  410. filechanges.add_change(ch);
  411. } else {
  412. char *last = NULL;
  413. char *add;
  414. size_t l;
  415. if (ch.add)
  416. last = ch.add + ch.add_len;
  417. l = strlen(buffer);
  418. add = add_text.add_easy(buffer, l, last);
  419. if (!add) {
  420. ch.add_len += l;
  421. ch.add_cnt++;
  422. } else {
  423. if (ch.add) {
  424. filechanges.add_change(ch);
  425. ch.del_cnt = 0;
  426. }
  427. ch.offset += ch.add_cnt;
  428. ch.add = add;
  429. ch.add_len = l;
  430. ch.add_cnt = 1;
  431. }
  432. }
  433. }
  434. } while(f.ReadLine(buffer, sizeof(buffer)));
  435. return true;
  436. }
  437. void write_diff(FileFd &f)
  438. {
  439. unsigned long long line = 0;
  440. std::list<struct Change>::reverse_iterator ch;
  441. for (ch = filechanges.rbegin(); ch != filechanges.rend(); ++ch) {
  442. line += ch->offset + ch->del_cnt;
  443. }
  444. for (ch = filechanges.rbegin(); ch != filechanges.rend(); ++ch) {
  445. std::list<struct Change>::reverse_iterator mg_i, mg_e = ch;
  446. while (ch->del_cnt == 0 && ch->offset == 0)
  447. {
  448. ++ch;
  449. if (unlikely(ch == filechanges.rend()))
  450. return;
  451. }
  452. line -= ch->del_cnt;
  453. std::string buf;
  454. if (ch->add_cnt > 0) {
  455. if (ch->del_cnt == 0) {
  456. strprintf(buf, "%llua\n", line);
  457. } else if (ch->del_cnt == 1) {
  458. strprintf(buf, "%lluc\n", line+1);
  459. } else {
  460. strprintf(buf, "%llu,%lluc\n", line+1, line+ch->del_cnt);
  461. }
  462. f.Write(buf.c_str(), buf.length());
  463. mg_i = ch;
  464. do {
  465. dump_mem(f, mg_i->add, mg_i->add_len, NULL);
  466. } while (mg_i-- != mg_e);
  467. buf = ".\n";
  468. f.Write(buf.c_str(), buf.length());
  469. } else if (ch->del_cnt == 1) {
  470. strprintf(buf, "%llud\n", line+1);
  471. f.Write(buf.c_str(), buf.length());
  472. } else if (ch->del_cnt > 1) {
  473. strprintf(buf, "%llu,%llud\n", line+1, line+ch->del_cnt);
  474. f.Write(buf.c_str(), buf.length());
  475. }
  476. line -= ch->offset;
  477. }
  478. }
  479. void apply_against_file(FileFd &out, FileFd &in,
  480. Hashes * const start_hash = nullptr, Hashes * const end_hash = nullptr)
  481. {
  482. std::list<struct Change>::iterator ch;
  483. for (ch = filechanges.begin(); ch != filechanges.end(); ++ch) {
  484. dump_lines(out, in, ch->offset, start_hash, end_hash);
  485. skip_lines(in, ch->del_cnt, start_hash);
  486. if (ch->add_len != 0)
  487. dump_mem(out, ch->add, ch->add_len, end_hash);
  488. }
  489. dump_rest(out, in, start_hash, end_hash);
  490. out.Flush();
  491. }
  492. };
  493. class RredMethod : public aptMethod {
  494. private:
  495. bool Debug;
  496. struct PDiffFile {
  497. std::string FileName;
  498. HashStringList ExpectedHashes;
  499. PDiffFile(std::string const &FileName, HashStringList const &ExpectedHashes) :
  500. FileName(FileName), ExpectedHashes(ExpectedHashes) {}
  501. };
  502. HashStringList ReadExpectedHashesForPatch(unsigned int const patch, std::string const &Message)
  503. {
  504. HashStringList ExpectedHashes;
  505. for (char const * const * type = HashString::SupportedHashes(); *type != NULL; ++type)
  506. {
  507. std::string tagname;
  508. strprintf(tagname, "Patch-%d-%s-Hash", patch, *type);
  509. std::string const hashsum = LookupTag(Message, tagname.c_str());
  510. if (hashsum.empty() == false)
  511. ExpectedHashes.push_back(HashString(*type, hashsum));
  512. }
  513. return ExpectedHashes;
  514. }
  515. protected:
  516. virtual bool URIAcquire(std::string const &Message, FetchItem *Itm) APT_OVERRIDE {
  517. Debug = DebugEnabled();
  518. URI Get = Itm->Uri;
  519. std::string Path = Get.Host + Get.Path; // rred:/path - no host
  520. FetchResult Res;
  521. Res.Filename = Itm->DestFile;
  522. if (Itm->Uri.empty())
  523. {
  524. Path = Itm->DestFile;
  525. Itm->DestFile.append(".result");
  526. } else
  527. URIStart(Res);
  528. std::vector<PDiffFile> patchfiles;
  529. Patch patch;
  530. HashStringList StartHashes;
  531. for (char const * const * type = HashString::SupportedHashes(); *type != nullptr; ++type)
  532. {
  533. std::string tagname;
  534. strprintf(tagname, "Start-%s-Hash", *type);
  535. std::string const hashsum = LookupTag(Message, tagname.c_str());
  536. if (hashsum.empty() == false)
  537. StartHashes.push_back(HashString(*type, hashsum));
  538. }
  539. if (FileExists(Path + ".ed") == true)
  540. {
  541. HashStringList const ExpectedHashes = ReadExpectedHashesForPatch(0, Message);
  542. std::string const FileName = Path + ".ed";
  543. if (ExpectedHashes.usable() == false)
  544. return _error->Error("No hashes found for uncompressed patch: %s", FileName.c_str());
  545. patchfiles.push_back(PDiffFile(FileName, ExpectedHashes));
  546. }
  547. else
  548. {
  549. _error->PushToStack();
  550. std::vector<std::string> patches = GetListOfFilesInDir(flNotFile(Path), "gz", true, false);
  551. _error->RevertToStack();
  552. std::string const baseName = Path + ".ed.";
  553. unsigned int seen_patches = 0;
  554. for (std::vector<std::string>::const_iterator p = patches.begin();
  555. p != patches.end(); ++p)
  556. {
  557. if (p->compare(0, baseName.length(), baseName) == 0)
  558. {
  559. HashStringList const ExpectedHashes = ReadExpectedHashesForPatch(seen_patches, Message);
  560. if (ExpectedHashes.usable() == false)
  561. return _error->Error("No hashes found for uncompressed patch %d: %s", seen_patches, p->c_str());
  562. patchfiles.push_back(PDiffFile(*p, ExpectedHashes));
  563. ++seen_patches;
  564. }
  565. }
  566. }
  567. std::string patch_name;
  568. for (std::vector<PDiffFile>::iterator I = patchfiles.begin();
  569. I != patchfiles.end();
  570. ++I)
  571. {
  572. patch_name = I->FileName;
  573. if (Debug == true)
  574. std::clog << "Patching " << Path << " with " << patch_name
  575. << std::endl;
  576. FileFd p;
  577. Hashes patch_hash(I->ExpectedHashes);
  578. // all patches are compressed, even if the name doesn't reflect it
  579. if (p.Open(patch_name, FileFd::ReadOnly, FileFd::Gzip) == false ||
  580. patch.read_diff(p, &patch_hash) == false)
  581. {
  582. _error->DumpErrors(std::cerr, GlobalError::DEBUG, false);
  583. return false;
  584. }
  585. p.Close();
  586. HashStringList const hsl = patch_hash.GetHashStringList();
  587. if (hsl != I->ExpectedHashes)
  588. return _error->Error("Hash Sum mismatch for uncompressed patch %s", patch_name.c_str());
  589. }
  590. if (Debug == true)
  591. std::clog << "Applying patches against " << Path
  592. << " and writing results to " << Itm->DestFile
  593. << std::endl;
  594. FileFd inp, out;
  595. if (inp.Open(Path, FileFd::ReadOnly, FileFd::Extension) == false)
  596. {
  597. if (Debug == true)
  598. std::clog << "FAILED to open inp " << Path << std::endl;
  599. return _error->Error("Failed to open inp %s", Path.c_str());
  600. }
  601. if (out.Open(Itm->DestFile, FileFd::WriteOnly | FileFd::Create | FileFd::Empty | FileFd::BufferedWrite, FileFd::Extension) == false)
  602. {
  603. if (Debug == true)
  604. std::clog << "FAILED to open out " << Itm->DestFile << std::endl;
  605. return _error->Error("Failed to open out %s", Itm->DestFile.c_str());
  606. }
  607. Hashes end_hash(Itm->ExpectedHashes);
  608. if (StartHashes.usable())
  609. {
  610. Hashes start_hash(StartHashes);
  611. patch.apply_against_file(out, inp, &start_hash, &end_hash);
  612. if (start_hash.GetHashStringList() != StartHashes)
  613. _error->Error("The input file hadn't the expected hash!");
  614. }
  615. else
  616. patch.apply_against_file(out, inp, nullptr, &end_hash);
  617. out.Close();
  618. inp.Close();
  619. if (_error->PendingError() == true) {
  620. if (Debug == true)
  621. std::clog << "FAILED to read or write files" << std::endl;
  622. return false;
  623. }
  624. if (Debug == true) {
  625. std::clog << "rred: finished file patching of " << Path << "." << std::endl;
  626. }
  627. struct stat bufbase, bufpatch;
  628. if (stat(Path.c_str(), &bufbase) != 0 ||
  629. stat(patch_name.c_str(), &bufpatch) != 0)
  630. return _error->Errno("stat", _("Failed to stat %s"), Path.c_str());
  631. struct timeval times[2];
  632. times[0].tv_sec = bufbase.st_atime;
  633. times[1].tv_sec = bufpatch.st_mtime;
  634. times[0].tv_usec = times[1].tv_usec = 0;
  635. if (utimes(Itm->DestFile.c_str(), times) != 0)
  636. return _error->Errno("utimes",_("Failed to set modification time"));
  637. if (stat(Itm->DestFile.c_str(), &bufbase) != 0)
  638. return _error->Errno("stat", _("Failed to stat %s"), Itm->DestFile.c_str());
  639. Res.LastModified = bufbase.st_mtime;
  640. Res.Size = bufbase.st_size;
  641. Res.TakeHashes(end_hash);
  642. URIDone(Res);
  643. return true;
  644. }
  645. public:
  646. RredMethod() : aptMethod("rred", "2.0", SendConfig), Debug(false)
  647. {
  648. SeccompFlags = aptMethod::BASE | aptMethod::DIRECTORY;
  649. }
  650. };
  651. int main(int argc, char **argv)
  652. {
  653. int i;
  654. bool just_diff = true;
  655. bool test = false;
  656. Patch patch;
  657. if (argc <= 1) {
  658. return RredMethod().Run();
  659. }
  660. // Usage: rred -t input output diff ...
  661. if (argc > 1 && strcmp(argv[1], "-t") == 0) {
  662. // Read config files so we see compressors.
  663. pkgInitConfig(*_config);
  664. just_diff = false;
  665. test = true;
  666. i = 4;
  667. } else if (argc > 1 && strcmp(argv[1], "-f") == 0) {
  668. just_diff = false;
  669. i = 2;
  670. } else {
  671. i = 1;
  672. }
  673. for (; i < argc; i++) {
  674. FileFd p;
  675. if (p.Open(argv[i], FileFd::ReadOnly) == false) {
  676. _error->DumpErrors(std::cerr);
  677. exit(1);
  678. }
  679. if (patch.read_diff(p, NULL) == false)
  680. {
  681. _error->DumpErrors(std::cerr);
  682. exit(2);
  683. }
  684. }
  685. if (test) {
  686. FileFd out, inp;
  687. std::cerr << "Patching " << argv[2] << " into " << argv[3] << "\n";
  688. inp.Open(argv[2], FileFd::ReadOnly,FileFd::Extension);
  689. out.Open(argv[3], FileFd::WriteOnly | FileFd::Create | FileFd::Empty | FileFd::BufferedWrite, FileFd::Extension);
  690. patch.apply_against_file(out, inp);
  691. out.Close();
  692. } else if (just_diff) {
  693. FileFd out;
  694. out.OpenDescriptor(STDOUT_FILENO, FileFd::WriteOnly | FileFd::Create);
  695. patch.write_diff(out);
  696. out.Close();
  697. } else {
  698. FileFd out, inp;
  699. out.OpenDescriptor(STDOUT_FILENO, FileFd::WriteOnly | FileFd::Create | FileFd::BufferedWrite);
  700. inp.OpenDescriptor(STDIN_FILENO, FileFd::ReadOnly);
  701. patch.apply_against_file(out, inp);
  702. out.Close();
  703. }
  704. return 0;
  705. }