defaria.com/blogs/Status/2005/11/check-attachmen.html

   1 <!DOCTYPE html>
   2 <html lang="en-us" itemscope itemtype="http://schema.org/Article">
   3   <head>
   4     <meta charset="utf-8">
   5     <meta name="description" content=" Worked out a plan for how to perform the PQA Merge this weekend Concerned about the size discrepancy in the database I wrote a little utility, check_attachments, to total up the size of all attachments in all the databases....">
   6     <meta name="generator" content="Movable Type 5.2.3">
   7     <title>check_attachments - Status</title>
   8     <link rel="alternate" type="application/atom+xml" title="Recent Entries" href="http://defaria.com/blogs/Status/atom.xml">
   9     <link rel="canonical" href="http://defaria.com/blogs/Status/2005/11/check-attachmen.html">
  10     <meta name="viewport" content="width=device-width,initial-scale=1">
  11     <link rel="stylesheet" href="http://defaria.com/blogs/Status/styles.css">
  12     <!--[if lt IE 9]>
  13     <link rel="stylesheet" href="http://defaria.com/blogs/Status/styles_ie.css">
  14     <script src="/mt/mt-static/support/theme_static/rainier/js/html5shiv.js"></script>
  15     <![endif]-->
  16
  17     <link rel="start" href="http://defaria.com/blogs/Status/">
  18
  19     <link rel="prev" href="http://defaria.com/blogs/Status/2005/11/bin-merge.html" title="bin_merge">
  20     <link rel="next" href="http://defaria.com/blogs/Status/2005/11/more-on-attachm.html" title="More on attachments">
  21     <!-- Open Graph Protocol -->
  22     <meta property="og:type" content="article">
  23     <meta property="og:locale" content="en-us">
  24     <meta property="og:title" content="check_attachments">
  25     <meta property="og:url" content="http://defaria.com/blogs/Status/2005/11/check-attachmen.html">
  26     <meta property="og:description" content=" Worked out a plan for how to perform the PQA Merge this weekend Concerned about the size discrepancy in the database I wrote a little utility, check_attachments, to total up the size of all attachments in all the databases....">
  27     <meta property="og:site_name" content="Status">
  28     <meta property="og:image" content="/mt/mt-static/support/theme_static/rainier/img/siteicon-sample.png">
  29     <!-- Metadata -->
  30     <meta itemprop="description" content=" Worked out a plan for how to perform the PQA Merge this weekend Concerned about the size discrepancy in the database I wrote a little utility, check_attachments, to total up the size of all attachments in all the databases....">
  31     <link itemprop="url" href="http://defaria.com/blogs/Status/2005/11/check-attachmen.html">
  32     <link itemprop="image" href="/mt/mt-static/support/theme_static/rainier/img/siteicon-sample.png">
  33
  34   </head>
  35   <body>
  36     <div id="container">
  37       <div id="container-inner">
  38         <header id="header" role="banner">
  39           <div id="header-inner">
  40             <div id="header-content">
  41               <h1>
  42                 <a href="http://defaria.com/blogs/Status/">
  43
  44                   Status
  45
  46                 </a>
  47               </h1>
  48
  49             </div>
  50
  51             <nav role="navigation">
  52           <ul>
  53             <li><a href="http://defaria.com/blogs/Status/">Home</a></li>
  54
  55
  56           </ul>
  57         </nav>
  58
  59           </div>
  60         </header>
  61         <div id="content">
  62           <div id="content-inner">
  63             <ul class="breadcrumb breadcrumb-list">
  64               <li class="breadcrumb-list-item"><a href="http://defaria.com/blogs/Status/">Home</a></li>
  65               <li class="breadcrumb-list-item">check_attachments</li>
  66             </ul>
  67             <div id="individual-main" class="main" role="main">
  68               <article id="entry-1734" class="entry entry-asset asset hentry">
  69                 <div class="asset-header">
  70                   <h2 itemprop="name" class="asset-name entry-title">check_attachments</h2>
  71                   <footer class="asset-meta">
  72                     <ul class="asset-meta-list">
  73                       <li class="asset-meta-list-item">Posted on <time datetime="2005-11-07T18:54:14-08:00" itemprop="datePublished">November  7, 2005</time></li>
  74                       <li class="asset-meta-list-item">by <span class="author entry-author vcard"></span></li>
  75
  76
  77                       <li class="asset-meta-list-item">in <a itemprop="articleSection" rel="tag" href="http://defaria.com/blogs/Status/pqa/">PQA</a></li>
  78
  79
  80                    </ul>
  81                 </footer>
  82                 </div>
  83                 <div class="entry-content asset-content" itemprop="articleBody">
  84                   <ul>
  85   <li>Worked out a plan for how to perform the PQA Merge this weekend</li>
  86
  87   <li>Concerned about the size discrepancy in the database I wrote a little utility, check_attachments, to total up the size of all attachments in all the databases. This turned up a bug in pqamerge which I fixed. Reperforming pqamerge</li>
  88
  89   <li>There is a problem with using clearprompt to prompt for list input for bin_merge - You can only have one line of prompting text and only 50 characters in that line. It's gonna be hard to describe a binary merge situation in 50 characters or less. Investigated making a PerlTk list dialog. In theory it can be done as ccperl does support PerlTk.</li>
  90
  91   <li>Discussed with Naga the <i>email link</i> issue with PQA</li>
  92 </ul>
  93                   <h3>TransferAttachments</h3>
  94
  95 <p>I knew there was a good reason why this was eating at me. There is indeed a bug. My attachments transfer routine was not getting called at all for Prod! I wrote a small Perl script to check this and it yielded:<p>
  96
  97 <div class="code"><pre>
  98 $ cqperl check_attachments
  99 Totaling attachments in TO...
 100 Totaling</span> attachments in Prod...
 101 Totaling</span> attachments in Cont...
 102 Total attachment size for TO = 272253137
 103 Total attachment size for Prod = 2683569547
 104 Total attachment size for Cont = 272253137
 105 </pre></div>
 106
 107 <p>Notice that the size of the attachments for TO matches the size of attachments for Cont! IOW no Prod attachments got transferred at all!</p>
 108
 109 <p>Turns out I was only calling TransferAttachments when the field name was AttachmentsBRCM (and TransferAttachments then did all the attachments - both Attachments and AttachmentsBRCM). However, in Prod the field is named AttachmentBRCM - note the
 110 singularity here! I tell ya my eyes are going.</p>
 111
 112 <p>I'm changing my code not be dependent on the field name and to just call TransferAttachments for each record. TransferAttachments doesn't need to know the field name - it just does all of them.</p>
 113
 114 <p>Unfortunately this means that I have to delete all of the records currently in the Cont database on p4test. And it also means that the approximate 4 hour running time will probably increase. However this does explain the discrepancy in the database sizes.</p>
 115
 116 <h3>Out of Disk Space</h3>
 117
 118 Shivdutt Jha wrote:
 119 <blockquote type=cite>
 120 Good job Andrew, another mystery is solved.
 121 </blockquote>
 122
 123 <p>What is more telling is that we have 2.6 Gig of attachments!</p>
 124
 125 <p>In any event my merge failed with out of disk space. The pqamerge script uses the current directory to temporarily hold the attachments as the are transfered from one DB to the other. Turns out that Prod00010818 has two very large attachments, one 328485466 bytes and another 219095209 bytes. Unfortunately I was running in ~vobadm/My Documents and that disk filled. So another lesson learned - run pqamerge in a directory on a disk with lots of disk space! Alas this means I have to start this process over again (I'm trying to get a clean run and a timing of how long to expect the merge to run. pqamerge outputs how long it takes to do it work).</p>
 126                 </div>
 127                 <nav class="page-navigation entry-navigation pagination content-nav">
 128                   <ul class="page-navigation-list">
 129
 130                     <li class="page-navigation-list-item page-navigation-prev"><a rel="prev" href="http://defaria.com/blogs/Status/2005/11/bin-merge.html" title="bin_merge">Previous entry</a></li>
 131
 132
 133                     <li class="page-navigation-list-item page-navigation-next"><a rel="next" href="http://defaria.com/blogs/Status/2005/11/more-on-attachm.html" title="More on attachments">Next entry</a></li>
 134
 135                   </ul>
 136                 </nav>
 137                 <!--
 138 <aside id="zenback" class="zenback feedback">
 139   Please paste Zenback script code here.
 140 </aside>
 141 -->
 142
 143
 144               </article>
 145             </div>
 146             <aside class="widgets related" role="complementary">
 147               <nav class="widget-search widget">
 148   <div class="widget-content">
 149     <form method="get" id="search" action="http://defaria.com/mt/mt-search.cgi">
 150       <div>
 151         <input type="text" name="search" value="" placeholder="Search...">
 152
 153         <input type="hidden" name="IncludeBlogs" value="8">
 154
 155         <input type="hidden" name="limit" value="20">
 156         <button type="submit" name="button">
 157           <img alt="Search" src="/mt/mt-static/support/theme_static/rainier/img/search-icon.png">
 158         </button>
 159       </div>
 160     </form>
 161   </div>
 162 </nav>
 163 <nav class="widget-archive-category widget">
 164   <h3 class="widget-header">Categories</h3>
 165   <div class="widget-content">
 166
 167
 168     <ul class="widget-list">
 169
 170
 171       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/ameriquest/">Ameriquest (99)</a>
 172
 173
 174       </li>
 175
 176
 177
 178
 179       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/audience/">Audience (3)</a>
 180
 181
 182       </li>
 183
 184
 185
 186
 187       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/broadcom/">Broadcom (76)</a>
 188
 189
 190       </li>
 191
 192
 193
 194
 195       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/gpdb/">GPDB (35)</a>
 196
 197
 198       </li>
 199
 200
 201
 202
 203       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/general-dynamics/">General Dynamics (61)</a>
 204
 205
 206       </li>
 207
 208
 209
 210
 211       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/general-electric/">General Electric (13)</a>
 212
 213
 214       </li>
 215
 216
 217
 218
 219       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/hewlett-packard/">Hewlett Packard (13)</a>
 220
 221
 222       </li>
 223
 224
 225
 226
 227       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/lynuxworks/">LynuxWorks (162)</a>
 228
 229
 230       </li>
 231
 232
 233
 234
 235       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/pqa/">PQA (35)</a>
 236
 237
 238       </li>
 239
 240
 241
 242
 243       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/salira/">Salira (79)</a>
 244
 245
 246       </li>
 247
 248
 249
 250
 251       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/tellabs/">Tellabs (2)</a>
 252
 253
 254       </li>
 255
 256
 257
 258
 259       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/texas-instruments/">Texas Instruments (31)</a>
 260
 261
 262       </li>
 263
 264     </ul>
 265
 266
 267   </div>
 268 </nav>
 269
 270
 271 <nav class="widget-archive-dropdown widget">
 272   <h3 class="widget-header">Archives</h3>
 273   <div class="widget-content">
 274     <select>
 275       <option>Select a Month...</option>
 276
 277       <option value="http://defaria.com/blogs/Status/2016/02/">February 2016</option>
 278
 279
 280
 281       <option value="http://defaria.com/blogs/Status/2014/09/">September 2014</option>
 282
 283
 284
 285       <option value="http://defaria.com/blogs/Status/2014/04/">April 2014</option>
 286
 287
 288
 289       <option value="http://defaria.com/blogs/Status/2014/03/">March 2014</option>
 290
 291
 292
 293       <option value="http://defaria.com/blogs/Status/2013/02/">February 2013</option>
 294
 295
 296
 297       <option value="http://defaria.com/blogs/Status/2012/09/">September 2012</option>
 298
 299
 300
 301       <option value="http://defaria.com/blogs/Status/2012/08/">August 2012</option>
 302
 303
 304
 305       <option value="http://defaria.com/blogs/Status/2012/05/">May 2012</option>
 306
 307
 308
 309       <option value="http://defaria.com/blogs/Status/2012/04/">April 2012</option>
 310
 311
 312
 313       <option value="http://defaria.com/blogs/Status/2012/02/">February 2012</option>
 314
 315
 316
 317       <option value="http://defaria.com/blogs/Status/2012/01/">January 2012</option>
 318
 319
 320
 321       <option value="http://defaria.com/blogs/Status/2011/10/">October 2011</option>
 322
 323
 324
 325       <option value="http://defaria.com/blogs/Status/2011/07/">July 2011</option>
 326
 327
 328
 329       <option value="http://defaria.com/blogs/Status/2010/09/">September 2010</option>
 330
 331
 332
 333       <option value="http://defaria.com/blogs/Status/2010/08/">August 2010</option>
 334
 335
 336
 337       <option value="http://defaria.com/blogs/Status/2010/04/">April 2010</option>
 338
 339
 340
 341       <option value="http://defaria.com/blogs/Status/2010/03/">March 2010</option>
 342
 343
 344
 345       <option value="http://defaria.com/blogs/Status/2010/02/">February 2010</option>
 346
 347
 348
 349       <option value="http://defaria.com/blogs/Status/2009/05/">May 2009</option>
 350
 351
 352
 353       <option value="http://defaria.com/blogs/Status/2009/04/">April 2009</option>
 354
 355
 356
 357       <option value="http://defaria.com/blogs/Status/2008/07/">July 2008</option>
 358
 359
 360
 361       <option value="http://defaria.com/blogs/Status/2008/05/">May 2008</option>
 362
 363
 364
 365       <option value="http://defaria.com/blogs/Status/2008/04/">April 2008</option>
 366
 367
 368
 369       <option value="http://defaria.com/blogs/Status/2008/03/">March 2008</option>
 370
 371
 372
 373       <option value="http://defaria.com/blogs/Status/2008/02/">February 2008</option>
 374
 375
 376
 377       <option value="http://defaria.com/blogs/Status/2008/01/">January 2008</option>
 378
 379
 380
 381       <option value="http://defaria.com/blogs/Status/2007/12/">December 2007</option>
 382
 383
 384
 385       <option value="http://defaria.com/blogs/Status/2007/11/">November 2007</option>
 386
 387
 388
 389       <option value="http://defaria.com/blogs/Status/2007/10/">October 2007</option>
 390
 391
 392
 393       <option value="http://defaria.com/blogs/Status/2007/09/">September 2007</option>
 394
 395
 396
 397       <option value="http://defaria.com/blogs/Status/2007/08/">August 2007</option>
 398
 399
 400
 401       <option value="http://defaria.com/blogs/Status/2007/07/">July 2007</option>
 402
 403
 404
 405       <option value="http://defaria.com/blogs/Status/2007/06/">June 2007</option>
 406
 407
 408
 409       <option value="http://defaria.com/blogs/Status/2007/05/">May 2007</option>
 410
 411
 412
 413       <option value="http://defaria.com/blogs/Status/2007/04/">April 2007</option>
 414
 415
 416
 417       <option value="http://defaria.com/blogs/Status/2007/03/">March 2007</option>
 418
 419
 420
 421       <option value="http://defaria.com/blogs/Status/2007/01/">January 2007</option>
 422
 423
 424
 425       <option value="http://defaria.com/blogs/Status/2006/12/">December 2006</option>
 426
 427
 428
 429       <option value="http://defaria.com/blogs/Status/2006/11/">November 2006</option>
 430
 431
 432
 433       <option value="http://defaria.com/blogs/Status/2006/10/">October 2006</option>
 434
 435
 436
 437       <option value="http://defaria.com/blogs/Status/2006/09/">September 2006</option>
 438
 439
 440
 441       <option value="http://defaria.com/blogs/Status/2006/07/">July 2006</option>
 442
 443
 444
 445       <option value="http://defaria.com/blogs/Status/2006/06/">June 2006</option>
 446
 447
 448
 449       <option value="http://defaria.com/blogs/Status/2006/05/">May 2006</option>
 450
 451
 452
 453       <option value="http://defaria.com/blogs/Status/2006/04/">April 2006</option>
 454
 455
 456
 457       <option value="http://defaria.com/blogs/Status/2006/03/">March 2006</option>
 458
 459
 460
 461       <option value="http://defaria.com/blogs/Status/2006/02/">February 2006</option>
 462
 463
 464
 465       <option value="http://defaria.com/blogs/Status/2006/01/">January 2006</option>
 466
 467
 468
 469       <option value="http://defaria.com/blogs/Status/2005/12/">December 2005</option>
 470
 471
 472
 473       <option value="http://defaria.com/blogs/Status/2005/11/">November 2005</option>
 474
 475
 476
 477       <option value="http://defaria.com/blogs/Status/2005/10/">October 2005</option>
 478
 479
 480
 481       <option value="http://defaria.com/blogs/Status/2005/09/">September 2005</option>
 482
 483
 484
 485       <option value="http://defaria.com/blogs/Status/2005/08/">August 2005</option>
 486
 487
 488
 489       <option value="http://defaria.com/blogs/Status/2005/07/">July 2005</option>
 490
 491
 492
 493       <option value="http://defaria.com/blogs/Status/2005/06/">June 2005</option>
 494
 495
 496
 497       <option value="http://defaria.com/blogs/Status/2005/05/">May 2005</option>
 498
 499
 500
 501       <option value="http://defaria.com/blogs/Status/2005/04/">April 2005</option>
 502
 503
 504
 505       <option value="http://defaria.com/blogs/Status/2005/03/">March 2005</option>
 506
 507
 508
 509       <option value="http://defaria.com/blogs/Status/2005/02/">February 2005</option>
 510
 511
 512
 513       <option value="http://defaria.com/blogs/Status/2005/01/">January 2005</option>
 514
 515
 516
 517       <option value="http://defaria.com/blogs/Status/2004/12/">December 2004</option>
 518
 519
 520
 521       <option value="http://defaria.com/blogs/Status/2004/09/">September 2004</option>
 522
 523
 524
 525       <option value="http://defaria.com/blogs/Status/2004/08/">August 2004</option>
 526
 527
 528
 529       <option value="http://defaria.com/blogs/Status/2004/07/">July 2004</option>
 530
 531
 532
 533       <option value="http://defaria.com/blogs/Status/2004/06/">June 2004</option>
 534
 535
 536
 537       <option value="http://defaria.com/blogs/Status/2004/05/">May 2004</option>
 538
 539
 540
 541       <option value="http://defaria.com/blogs/Status/2004/04/">April 2004</option>
 542
 543
 544
 545       <option value="http://defaria.com/blogs/Status/2004/03/">March 2004</option>
 546
 547
 548
 549       <option value="http://defaria.com/blogs/Status/2004/02/">February 2004</option>
 550
 551
 552
 553       <option value="http://defaria.com/blogs/Status/2004/01/">January 2004</option>
 554
 555
 556
 557       <option value="http://defaria.com/blogs/Status/2003/12/">December 2003</option>
 558
 559
 560
 561       <option value="http://defaria.com/blogs/Status/2003/11/">November 2003</option>
 562
 563     </select>
 564   </div>
 565 </nav>
 566
 567
 568
 569 <div class="widget-syndication widget section">
 570   <div class="widget-content">
 571     <p><img src="http://defaria.com/mt/mt-static/images/status_icons/feed.gif" alt="Subscribe to feed" width="9" height="9" /> <a href="http://defaria.com/blogs/Status/atom.xml">Subscribe to this blog's feed</a></p>
 572
 573   </div>
 574 </div>
 575
 576             </aside>
 577           </div>
 578         </div>
 579         <footer id="footer" role="contentinfo">
 580           <div id="footer-inner">
 581             <div id="footer-content">
 582   <nav role="navigation">
 583           <ul>
 584             <li><a href="http://defaria.com/blogs/Status/">Home</a></li>
 585
 586
 587           </ul>
 588         </nav>
 589
 590   <p class="license">&copy; Copyright 2016.</p>
 591   <p class="poweredby">Powered by <a href="http://www.movabletype.org/">Movable Type</a></p>
 592 </div>
 593           </div>
 594         </footer>
 595       </div>
 596     </div>
 597     <script src="http://defaria.com/mt/mt-static/jquery/jquery.min.js"></script>
 598     <script src="http://defaria.com/blogs/Status/mt-theme-scale2.js"></script>
 599   </body>
 600 </html>