defaria.com/blogs/Status/2006/11/gpdb-add-projec-3.html

   1 <!DOCTYPE html>
   2 <html lang="en-us" itemscope itemtype="http://schema.org/Article">
   3   <head>
   4     <meta charset="utf-8">
   5     <meta name="description" content=" Looked into sites, site codes and differences between UK projects/domains and the current implementation of GPDB Cronned gpdb_add_project...">
   6     <meta name="generator" content="Movable Type 5.2.3">
   7     <title>gpdb_add_project cronned - Status</title>
   8     <link rel="alternate" type="application/atom+xml" title="Recent Entries" href="http://defaria.com/blogs/Status/atom.xml">
   9     <link rel="canonical" href="http://defaria.com/blogs/Status/2006/11/gpdb-add-projec-3.html">
  10     <meta name="viewport" content="width=device-width,initial-scale=1">
  11     <link rel="stylesheet" href="http://defaria.com/blogs/Status/styles.css">
  12     <!--[if lt IE 9]>
  13     <link rel="stylesheet" href="http://defaria.com/blogs/Status/styles_ie.css">
  14     <script src="/mt/mt-static/support/theme_static/rainier/js/html5shiv.js"></script>
  15     <![endif]-->
  16
  17     <link rel="start" href="http://defaria.com/blogs/Status/">
  18
  19     <link rel="prev" href="http://defaria.com/blogs/Status/2006/11/uk-gpdb.html" title="UK -&gt; GPDB">
  20     <link rel="next" href="http://defaria.com/blogs/Status/2006/11/sites-againcron.html" title="Sites again/Cron problems">
  21     <!-- Open Graph Protocol -->
  22     <meta property="og:type" content="article">
  23     <meta property="og:locale" content="en-us">
  24     <meta property="og:title" content="gpdb_add_project cronned">
  25     <meta property="og:url" content="http://defaria.com/blogs/Status/2006/11/gpdb-add-projec-3.html">
  26     <meta property="og:description" content=" Looked into sites, site codes and differences between UK projects/domains and the current implementation of GPDB Cronned gpdb_add_project...">
  27     <meta property="og:site_name" content="Status">
  28     <meta property="og:image" content="/mt/mt-static/support/theme_static/rainier/img/siteicon-sample.png">
  29     <!-- Metadata -->
  30     <meta itemprop="description" content=" Looked into sites, site codes and differences between UK projects/domains and the current implementation of GPDB Cronned gpdb_add_project...">
  31     <link itemprop="url" href="http://defaria.com/blogs/Status/2006/11/gpdb-add-projec-3.html">
  32     <link itemprop="image" href="/mt/mt-static/support/theme_static/rainier/img/siteicon-sample.png">
  33
  34   </head>
  35   <body>
  36     <div id="container">
  37       <div id="container-inner">
  38         <header id="header" role="banner">
  39           <div id="header-inner">
  40             <div id="header-content">
  41               <h1>
  42                 <a href="http://defaria.com/blogs/Status/">
  43
  44                   Status
  45
  46                 </a>
  47               </h1>
  48
  49             </div>
  50
  51             <nav role="navigation">
  52           <ul>
  53             <li><a href="http://defaria.com/blogs/Status/">Home</a></li>
  54
  55
  56           </ul>
  57         </nav>
  58
  59           </div>
  60         </header>
  61         <div id="content">
  62           <div id="content-inner">
  63             <ul class="breadcrumb breadcrumb-list">
  64               <li class="breadcrumb-list-item"><a href="http://defaria.com/blogs/Status/">Home</a></li>
  65               <li class="breadcrumb-list-item">gpdb_add_project cronned</li>
  66             </ul>
  67             <div id="individual-main" class="main" role="main">
  68               <article id="entry-1848" class="entry entry-asset asset hentry">
  69                 <div class="asset-header">
  70                   <h2 itemprop="name" class="asset-name entry-title">gpdb_add_project cronned</h2>
  71                   <footer class="asset-meta">
  72                     <ul class="asset-meta-list">
  73                       <li class="asset-meta-list-item">Posted on <time datetime="2006-11-14T10:58:26-08:00" itemprop="datePublished">November 14, 2006</time></li>
  74                       <li class="asset-meta-list-item">by <span class="author entry-author vcard"></span></li>
  75
  76
  77                       <li class="asset-meta-list-item">in <a itemprop="articleSection" rel="tag" href="http://defaria.com/blogs/Status/gpdb/">GPDB</a></li>
  78
  79
  80                    </ul>
  81                 </footer>
  82                 </div>
  83                 <div class="entry-content asset-content" itemprop="articleBody">
  84                   <ul>
  85   <li>Looked into sites, site codes and differences between UK projects/domains and the current implementation of GPDB</li>
  86
  87   <li>Cronned gpdb_add_project</li>
  88 </ul>
  89                   <h2>Cronned gpdb_add_project</h2>
  90
  91 <p>Donna Ducharme wrote:</p>
  92
  93 <blockquote type="cite">
  94   <p>GPDB is not current. This project CC2630_DS is in the dallas site registry but is not in GPDB. Have we missed something here?</p>
  95 </blockquote>
  96
  97 <h3>Short answer (AKA Executive Summary)</h3>
  98
  99 <p>It's there now. gpdb_add_project.pl wasn't regularly run. Now it is</p>
 100
 101 <h3>Long answer (AKA Engineering Notes)</h3>
 102
 103 <p>gpdb_add_project.pl, the script that attempts to add any new DesignSync projects to GPDB, is not run on a regular basis. Why is this? Well because in the past this was simply cron'ed into somebody's crontab. Then the guy left and this broke. The right way to do this is to eliminate dependencies to any employee or, as the case was here, contractor. We've been working on that.</p>
 104
 105 <p>First step was to get gpdb_add_project.pl functioning again. Being as it attempts to do all sites and that it used to do this by using rsh, this broke when the guy left. Why? Well because he was rsh'ing as himself and his account naturally, and rightfully, got disabled when he left. So David Kitch set about to create a generic user, gpdb, and get passwordless rsh login rights to all of the appropriate servers at the sites. Recently that was completed.</p>
 106
 107 <p>Meantime I was busy recoding gpdb_add_project.pl as was put into Clearcase and use strict and use warnings were strapped on but the code as checked in would not even compile! First task was to resolve the problems use strict and use warnings introduced.</p>
 108
 109 <p>Next was to resolve the incorrect utilization of rsh that gpdb_add_project.pl was doing It would do an rsh command and check the return status thinking that it was the return states of the command that rsh remotely executed. That is not the case! Rsh returns the status code of the rsh command itself, not the command remotely executed. Additionally it's pretty inefficient to constantly establish communications with another system via rsh, do one single command then tear down the whole remote channel only to do yet another rsh command shortly. Additionally we wish to move to the more secure and better ssh method in the future.</p>
 110
 111 <p>So I wrote Rexec.pm, a Perl module that creates an object and a connection to a remote system<sup>1</sup>. It has additional functionality to allow not only rsh access, but ssh and even telnet. It can access another username/password to attempt access with. While passwordless login is preferred and passwords are also handled. Expect is utilized to drive this. Also this gracefully degrades in that if a specific protocol is not specified then ssh is first tried, followed by rsh then finally telnet. Also, the connection is held in the object so that multiple command execution is quick and efficient. Finally it reliably returns the output from the remote command as well as it's status. Handles different shell styles (basically csh style shells and sh style shells) but must be informed ahead of time of which shell to expect. Downside, only handles "standard" prompts (generic users should always be configured with a standard/default  prompt).</p>
 112
 113 <ol>
 114   <li><small>Why didn't I use a CPAN module for this? Many reasons: It offered nothing above what gpdb_add_project.pl already did - i.e. it returned the status of rsh not the command rsh did. Also it didn't support being instantiated and remaining active for multiple commands. Finally it didn't support any other protocol than rsh so there was no preference for ssh and graceful degradation.</small></li>
 115 </ol>
 116
 117 <p>With Rexec.pm in place I set out to make gpdb_add_project.pl aware of it and to utilize it. Meantime David was hard at work getting gpdb passwordless login available at all sites. About a week or two ago gpdb_add_project.pl successfully interrogated the final site, Manchester, and updated GPDB of all known DesignSync projects at that time.</p>
 118
 119 <p>And just today I worked with Michael Tisdel to gain access to a gpdb login to Cashew, the system which will house gpdb's crontab and execute such scripts from there. So, IOW, I just got the ability to automate this and have done so.</p>
 120
 121 <p>Still we have no official release mechanism for this gpdb_add_project.pl yet so I just copied it to cashew:~gpdb and it's running from there. I had to additionally copy the Rexec.pm module there as well as the gpdb_site_list.txt which drives which sites to explore. This needs to change when we official release gpdb_add_project.pl.</p>
 122
 123 <p>There is much more work to do with gpdb_add_project.pl. Reporting is poor (it creates logs in ~gpdb/gpdb_add_project_logs), it does not properly handle nor alert people when a server found in DesignSync is not reachable (We have a GPDB CQ request for that one) and from what I can tell it's checking of IP ranges is not correct.</p>
 124
 125 <p>I've added the following to gpdb's crontab on cashew:</p>
 126
 127 <div class=code><pre>
 128 # <a href="mailto:Andrew@DeFaria.com:">Andrew@DeFaria.com:</a> Run gpdb_add_project.pl. This will add any new projects
 129 # from DesignSync to GPDB. Note that this runs from right here (~gpdb).This
 130 # is wrong and should be changed to run it from the "standard place".Problem
 131 # is - there is no "standard place" (yet).
 132 #
 133 # Also note that this script logs to ~/gpdb_add_project_logs. I'm not sure
 134 # who will read this or clean this up...
 135 00 00 * * * gpdb_add_project.pl -s gpdb_site_list.txt
 136 </pre></div>
 137
 138 <p>Again, there's lots of new stuff, enhancements and changes that gpdb_add_project.pl will have to support as we update GPDB to better support Clearcase and the changes to the database that that requires.</p>
 139
 140 <p><font color="#c0c0c0">End of "everything you wanted to know about gpdb_add_project.pl but were afraid to ask"....</font></p>
 141                 </div>
 142                 <nav class="page-navigation entry-navigation pagination content-nav">
 143                   <ul class="page-navigation-list">
 144
 145                     <li class="page-navigation-list-item page-navigation-prev"><a rel="prev" href="http://defaria.com/blogs/Status/2006/11/uk-gpdb.html" title="UK -&gt; GPDB">Previous entry</a></li>
 146
 147
 148                     <li class="page-navigation-list-item page-navigation-next"><a rel="next" href="http://defaria.com/blogs/Status/2006/11/sites-againcron.html" title="Sites again/Cron problems">Next entry</a></li>
 149
 150                   </ul>
 151                 </nav>
 152                 <!--
 153 <aside id="zenback" class="zenback feedback">
 154   Please paste Zenback script code here.
 155 </aside>
 156 -->
 157
 158
 159               </article>
 160             </div>
 161             <aside class="widgets related" role="complementary">
 162               <nav class="widget-search widget">
 163   <div class="widget-content">
 164     <form method="get" id="search" action="http://defaria.com/mt/mt-search.cgi">
 165       <div>
 166         <input type="text" name="search" value="" placeholder="Search...">
 167
 168         <input type="hidden" name="IncludeBlogs" value="8">
 169
 170         <input type="hidden" name="limit" value="20">
 171         <button type="submit" name="button">
 172           <img alt="Search" src="/mt/mt-static/support/theme_static/rainier/img/search-icon.png">
 173         </button>
 174       </div>
 175     </form>
 176   </div>
 177 </nav>
 178 <nav class="widget-archive-category widget">
 179   <h3 class="widget-header">Categories</h3>
 180   <div class="widget-content">
 181
 182
 183     <ul class="widget-list">
 184
 185
 186       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/ameriquest/">Ameriquest (99)</a>
 187
 188
 189       </li>
 190
 191
 192
 193
 194       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/audience/">Audience (3)</a>
 195
 196
 197       </li>
 198
 199
 200
 201
 202       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/broadcom/">Broadcom (76)</a>
 203
 204
 205       </li>
 206
 207
 208
 209
 210       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/gpdb/">GPDB (35)</a>
 211
 212
 213       </li>
 214
 215
 216
 217
 218       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/general-dynamics/">General Dynamics (61)</a>
 219
 220
 221       </li>
 222
 223
 224
 225
 226       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/general-electric/">General Electric (13)</a>
 227
 228
 229       </li>
 230
 231
 232
 233
 234       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/hewlett-packard/">Hewlett Packard (13)</a>
 235
 236
 237       </li>
 238
 239
 240
 241
 242       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/lynuxworks/">LynuxWorks (162)</a>
 243
 244
 245       </li>
 246
 247
 248
 249
 250       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/pqa/">PQA (35)</a>
 251
 252
 253       </li>
 254
 255
 256
 257
 258       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/salira/">Salira (79)</a>
 259
 260
 261       </li>
 262
 263
 264
 265
 266       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/tellabs/">Tellabs (2)</a>
 267
 268
 269       </li>
 270
 271
 272
 273
 274       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/texas-instruments/">Texas Instruments (31)</a>
 275
 276
 277       </li>
 278
 279     </ul>
 280
 281
 282   </div>
 283 </nav>
 284
 285
 286 <nav class="widget-archive-dropdown widget">
 287   <h3 class="widget-header">Archives</h3>
 288   <div class="widget-content">
 289     <select>
 290       <option>Select a Month...</option>
 291
 292       <option value="http://defaria.com/blogs/Status/2016/02/">February 2016</option>
 293
 294
 295
 296       <option value="http://defaria.com/blogs/Status/2014/09/">September 2014</option>
 297
 298
 299
 300       <option value="http://defaria.com/blogs/Status/2014/04/">April 2014</option>
 301
 302
 303
 304       <option value="http://defaria.com/blogs/Status/2014/03/">March 2014</option>
 305
 306
 307
 308       <option value="http://defaria.com/blogs/Status/2013/02/">February 2013</option>
 309
 310
 311
 312       <option value="http://defaria.com/blogs/Status/2012/09/">September 2012</option>
 313
 314
 315
 316       <option value="http://defaria.com/blogs/Status/2012/08/">August 2012</option>
 317
 318
 319
 320       <option value="http://defaria.com/blogs/Status/2012/05/">May 2012</option>
 321
 322
 323
 324       <option value="http://defaria.com/blogs/Status/2012/04/">April 2012</option>
 325
 326
 327
 328       <option value="http://defaria.com/blogs/Status/2012/02/">February 2012</option>
 329
 330
 331
 332       <option value="http://defaria.com/blogs/Status/2012/01/">January 2012</option>
 333
 334
 335
 336       <option value="http://defaria.com/blogs/Status/2011/10/">October 2011</option>
 337
 338
 339
 340       <option value="http://defaria.com/blogs/Status/2011/07/">July 2011</option>
 341
 342
 343
 344       <option value="http://defaria.com/blogs/Status/2010/09/">September 2010</option>
 345
 346
 347
 348       <option value="http://defaria.com/blogs/Status/2010/08/">August 2010</option>
 349
 350
 351
 352       <option value="http://defaria.com/blogs/Status/2010/04/">April 2010</option>
 353
 354
 355
 356       <option value="http://defaria.com/blogs/Status/2010/03/">March 2010</option>
 357
 358
 359
 360       <option value="http://defaria.com/blogs/Status/2010/02/">February 2010</option>
 361
 362
 363
 364       <option value="http://defaria.com/blogs/Status/2009/05/">May 2009</option>
 365
 366
 367
 368       <option value="http://defaria.com/blogs/Status/2009/04/">April 2009</option>
 369
 370
 371
 372       <option value="http://defaria.com/blogs/Status/2008/07/">July 2008</option>
 373
 374
 375
 376       <option value="http://defaria.com/blogs/Status/2008/05/">May 2008</option>
 377
 378
 379
 380       <option value="http://defaria.com/blogs/Status/2008/04/">April 2008</option>
 381
 382
 383
 384       <option value="http://defaria.com/blogs/Status/2008/03/">March 2008</option>
 385
 386
 387
 388       <option value="http://defaria.com/blogs/Status/2008/02/">February 2008</option>
 389
 390
 391
 392       <option value="http://defaria.com/blogs/Status/2008/01/">January 2008</option>
 393
 394
 395
 396       <option value="http://defaria.com/blogs/Status/2007/12/">December 2007</option>
 397
 398
 399
 400       <option value="http://defaria.com/blogs/Status/2007/11/">November 2007</option>
 401
 402
 403
 404       <option value="http://defaria.com/blogs/Status/2007/10/">October 2007</option>
 405
 406
 407
 408       <option value="http://defaria.com/blogs/Status/2007/09/">September 2007</option>
 409
 410
 411
 412       <option value="http://defaria.com/blogs/Status/2007/08/">August 2007</option>
 413
 414
 415
 416       <option value="http://defaria.com/blogs/Status/2007/07/">July 2007</option>
 417
 418
 419
 420       <option value="http://defaria.com/blogs/Status/2007/06/">June 2007</option>
 421
 422
 423
 424       <option value="http://defaria.com/blogs/Status/2007/05/">May 2007</option>
 425
 426
 427
 428       <option value="http://defaria.com/blogs/Status/2007/04/">April 2007</option>
 429
 430
 431
 432       <option value="http://defaria.com/blogs/Status/2007/03/">March 2007</option>
 433
 434
 435
 436       <option value="http://defaria.com/blogs/Status/2007/01/">January 2007</option>
 437
 438
 439
 440       <option value="http://defaria.com/blogs/Status/2006/12/">December 2006</option>
 441
 442
 443
 444       <option value="http://defaria.com/blogs/Status/2006/11/">November 2006</option>
 445
 446
 447
 448       <option value="http://defaria.com/blogs/Status/2006/10/">October 2006</option>
 449
 450
 451
 452       <option value="http://defaria.com/blogs/Status/2006/09/">September 2006</option>
 453
 454
 455
 456       <option value="http://defaria.com/blogs/Status/2006/07/">July 2006</option>
 457
 458
 459
 460       <option value="http://defaria.com/blogs/Status/2006/06/">June 2006</option>
 461
 462
 463
 464       <option value="http://defaria.com/blogs/Status/2006/05/">May 2006</option>
 465
 466
 467
 468       <option value="http://defaria.com/blogs/Status/2006/04/">April 2006</option>
 469
 470
 471
 472       <option value="http://defaria.com/blogs/Status/2006/03/">March 2006</option>
 473
 474
 475
 476       <option value="http://defaria.com/blogs/Status/2006/02/">February 2006</option>
 477
 478
 479
 480       <option value="http://defaria.com/blogs/Status/2006/01/">January 2006</option>
 481
 482
 483
 484       <option value="http://defaria.com/blogs/Status/2005/12/">December 2005</option>
 485
 486
 487
 488       <option value="http://defaria.com/blogs/Status/2005/11/">November 2005</option>
 489
 490
 491
 492       <option value="http://defaria.com/blogs/Status/2005/10/">October 2005</option>
 493
 494
 495
 496       <option value="http://defaria.com/blogs/Status/2005/09/">September 2005</option>
 497
 498
 499
 500       <option value="http://defaria.com/blogs/Status/2005/08/">August 2005</option>
 501
 502
 503
 504       <option value="http://defaria.com/blogs/Status/2005/07/">July 2005</option>
 505
 506
 507
 508       <option value="http://defaria.com/blogs/Status/2005/06/">June 2005</option>
 509
 510
 511
 512       <option value="http://defaria.com/blogs/Status/2005/05/">May 2005</option>
 513
 514
 515
 516       <option value="http://defaria.com/blogs/Status/2005/04/">April 2005</option>
 517
 518
 519
 520       <option value="http://defaria.com/blogs/Status/2005/03/">March 2005</option>
 521
 522
 523
 524       <option value="http://defaria.com/blogs/Status/2005/02/">February 2005</option>
 525
 526
 527
 528       <option value="http://defaria.com/blogs/Status/2005/01/">January 2005</option>
 529
 530
 531
 532       <option value="http://defaria.com/blogs/Status/2004/12/">December 2004</option>
 533
 534
 535
 536       <option value="http://defaria.com/blogs/Status/2004/09/">September 2004</option>
 537
 538
 539
 540       <option value="http://defaria.com/blogs/Status/2004/08/">August 2004</option>
 541
 542
 543
 544       <option value="http://defaria.com/blogs/Status/2004/07/">July 2004</option>
 545
 546
 547
 548       <option value="http://defaria.com/blogs/Status/2004/06/">June 2004</option>
 549
 550
 551
 552       <option value="http://defaria.com/blogs/Status/2004/05/">May 2004</option>
 553
 554
 555
 556       <option value="http://defaria.com/blogs/Status/2004/04/">April 2004</option>
 557
 558
 559
 560       <option value="http://defaria.com/blogs/Status/2004/03/">March 2004</option>
 561
 562
 563
 564       <option value="http://defaria.com/blogs/Status/2004/02/">February 2004</option>
 565
 566
 567
 568       <option value="http://defaria.com/blogs/Status/2004/01/">January 2004</option>
 569
 570
 571
 572       <option value="http://defaria.com/blogs/Status/2003/12/">December 2003</option>
 573
 574
 575
 576       <option value="http://defaria.com/blogs/Status/2003/11/">November 2003</option>
 577
 578     </select>
 579   </div>
 580 </nav>
 581
 582
 583
 584 <div class="widget-syndication widget section">
 585   <div class="widget-content">
 586     <p><img src="http://defaria.com/mt/mt-static/images/status_icons/feed.gif" alt="Subscribe to feed" width="9" height="9" /> <a href="http://defaria.com/blogs/Status/atom.xml">Subscribe to this blog's feed</a></p>
 587
 588   </div>
 589 </div>
 590
 591             </aside>
 592           </div>
 593         </div>
 594         <footer id="footer" role="contentinfo">
 595           <div id="footer-inner">
 596             <div id="footer-content">
 597   <nav role="navigation">
 598           <ul>
 599             <li><a href="http://defaria.com/blogs/Status/">Home</a></li>
 600
 601
 602           </ul>
 603         </nav>
 604
 605   <p class="license">&copy; Copyright 2016.</p>
 606   <p class="poweredby">Powered by <a href="http://www.movabletype.org/">Movable Type</a></p>
 607 </div>
 608           </div>
 609         </footer>
 610       </div>
 611     </div>
 612     <script src="http://defaria.com/mt/mt-static/jquery/jquery.min.js"></script>
 613     <script src="http://defaria.com/blogs/Status/mt-theme-scale2.js"></script>
 614   </body>
 615 </html>