defaria.com/blogs/Status/2005/12/code-pagecreate.html

   1 <!DOCTYPE html>
   2 <html lang="en-us" itemscope itemtype="http://schema.org/Article">
   3   <head>
   4     <meta charset="utf-8">
   5     <meta name="description" content=" Researched further code page issues Added Everest (Isreal0 team members to PQA Added -reuse_stream parm to create_dev_snapview.pl Handed a preliminary copy of bin_merge to Jennifer Submitted defect to IBM/Rational regarding CharacterSetValidation and Code Pages...">
   6     <meta name="generator" content="Movable Type 5.2.3">
   7     <title>Code Page/create_dev_snapview.pl - Status</title>
   8     <link rel="alternate" type="application/atom+xml" title="Recent Entries" href="http://defaria.com/blogs/Status/atom.xml">
   9     <link rel="canonical" href="http://defaria.com/blogs/Status/2005/12/code-pagecreate.html">
  10     <meta name="viewport" content="width=device-width,initial-scale=1">
  11     <link rel="stylesheet" href="http://defaria.com/blogs/Status/styles.css">
  12     <!--[if lt IE 9]>
  13     <link rel="stylesheet" href="http://defaria.com/blogs/Status/styles_ie.css">
  14     <script src="/mt/mt-static/support/theme_static/rainier/js/html5shiv.js"></script>
  15     <![endif]-->
  16
  17     <link rel="start" href="http://defaria.com/blogs/Status/">
  18
  19     <link rel="prev" href="http://defaria.com/blogs/Status/2005/11/post-2.html" title="Stgloc/Code Page">
  20     <link rel="next" href="http://defaria.com/blogs/Status/2005/12/scripts.html" title="Cleardiffmrg/bin_merge problems">
  21     <!-- Open Graph Protocol -->
  22     <meta property="og:type" content="article">
  23     <meta property="og:locale" content="en-us">
  24     <meta property="og:title" content="Code Page/create_dev_snapview.pl">
  25     <meta property="og:url" content="http://defaria.com/blogs/Status/2005/12/code-pagecreate.html">
  26     <meta property="og:description" content=" Researched further code page issues Added Everest (Isreal0 team members to PQA Added -reuse_stream parm to create_dev_snapview.pl Handed a preliminary copy of bin_merge to Jennifer Submitted defect to IBM/Rational regarding CharacterSetValidation and Code Pages...">
  27     <meta property="og:site_name" content="Status">
  28     <meta property="og:image" content="/mt/mt-static/support/theme_static/rainier/img/siteicon-sample.png">
  29     <!-- Metadata -->
  30     <meta itemprop="description" content=" Researched further code page issues Added Everest (Isreal0 team members to PQA Added -reuse_stream parm to create_dev_snapview.pl Handed a preliminary copy of bin_merge to Jennifer Submitted defect to IBM/Rational regarding CharacterSetValidation and Code Pages...">
  31     <link itemprop="url" href="http://defaria.com/blogs/Status/2005/12/code-pagecreate.html">
  32     <link itemprop="image" href="/mt/mt-static/support/theme_static/rainier/img/siteicon-sample.png">
  33
  34   </head>
  35   <body>
  36     <div id="container">
  37       <div id="container-inner">
  38         <header id="header" role="banner">
  39           <div id="header-inner">
  40             <div id="header-content">
  41               <h1>
  42                 <a href="http://defaria.com/blogs/Status/">
  43
  44                   Status
  45
  46                 </a>
  47               </h1>
  48
  49             </div>
  50
  51             <nav role="navigation">
  52           <ul>
  53             <li><a href="http://defaria.com/blogs/Status/">Home</a></li>
  54
  55
  56           </ul>
  57         </nav>
  58
  59           </div>
  60         </header>
  61         <div id="content">
  62           <div id="content-inner">
  63             <ul class="breadcrumb breadcrumb-list">
  64               <li class="breadcrumb-list-item"><a href="http://defaria.com/blogs/Status/">Home</a></li>
  65               <li class="breadcrumb-list-item">Code Page/create_dev_snapview.pl</li>
  66             </ul>
  67             <div id="individual-main" class="main" role="main">
  68               <article id="entry-1752" class="entry entry-asset asset hentry">
  69                 <div class="asset-header">
  70                   <h2 itemprop="name" class="asset-name entry-title">Code Page/create_dev_snapview.pl</h2>
  71                   <footer class="asset-meta">
  72                     <ul class="asset-meta-list">
  73                       <li class="asset-meta-list-item">Posted on <time datetime="2005-12-01T16:58:11-08:00" itemprop="datePublished">December  1, 2005</time></li>
  74                       <li class="asset-meta-list-item">by <span class="author entry-author vcard"></span></li>
  75
  76
  77                       <li class="asset-meta-list-item">in <a itemprop="articleSection" rel="tag" href="http://defaria.com/blogs/Status/broadcom/">Broadcom</a></li>
  78
  79
  80                    </ul>
  81                 </footer>
  82                 </div>
  83                 <div class="entry-content asset-content" itemprop="articleBody">
  84                   <ul>
  85   <li>Researched further code page issues</li>
  86
  87   <li>Added Everest (Isreal0 team members to PQA</li>
  88
  89   <li>Added -reuse_stream parm to create_dev_snapview.pl</li>
  90
  91   <li>Handed a preliminary copy of bin_merge to Jennifer</li>
  92
  93   <li>Submitted defect to IBM/Rational regarding CharacterSetValidation and Code Pages</li>
  94 </ul>
  95                   <h2>CharacterSetValidation Package is not that good</h2>
  96
  97 <p>We have a remaining issue where a user will attempt to update a defect and receive an error from the CharacterSetValidation package. I re-engineered CheckCodePage.pl to check through the Controller database but it didn't find anything. Investigated the CharacterSetValidation package a little more in depth - how does it validate characters? How does it do it differently than me?</p>
  98
  99 <p>Turns out that it merely does:</p>
 100
 101 <div class="code"><pre>
 102   if ($line =~ /[^\t\n\r -\177]/) {
 103     <i>error</i>
 104   }
 105 </pre></div>
 106
 107 <p>This odd regex seems to be merely checking to see if the characters are printable or not. I don't see how this would, for example, prevent a non US ASCII character like é from slipping through!</p>
 108
 109 <p>Narrowing it down a bit with my CheckCodePage.pl (which merely checks that characters are in the range of ordinal 0 - 127 (AKA the US ASCII character set) I figured out that the above regex was failing on characters such as ordinal 7 (Bell) and others. IOW characters that are technically in the US ASCII character set but that are not printable. Not sure how to resolve this problem yet...</p>
 110
 111 <h2>Submitted defect to IBM/Rational regarding CharacterSetValidation and CodePages</h2>
 112
 113 <p>Recently we upgraded our Clearquest database using 2003.06.15. With this version of Clearquest comes the usage of Code Pages. We choose to set our Code Page to US-ASCII. In upgrading our database we checked to ensure that all character data was within the US-ASCII character set (characters in the range of 0-127). Additionally we installed the CharacterSetValidation package as the Clearquest Administration Guide says:</p>
 114
 115 <blockquote>
 116   <p>The CharacterSetValidation package prevents clients running earlier versions of ClearQuest from entering data in a user record from a code page other than the data code page value of that database. If you do not apply the CharacterSetValidation package to your schemas, it is possible for users to enter unsupported data from the client and for data to be corrupted when modified on certain clients.</p>
 117 </blockquote>
 118
 119 <p>However it doesn't appear that the CharacterSetValidation package properly prevents unsupported data from being entered. Additionally it actually is causing us problems because characters that are valid US-ASCII are being flagged by the CharacterSetValidation package as unsupported.</p>
 120
 121 <p>US-ASCII is, according to <a href="http://en.wikipedia.org/wiki/ASCII:">http://en.wikipedia.org/wiki/ASCII</a>:</p>
 122
 123 <blockquote>
 124   <p>ASCII is, strictly, a seven-<a href="http://en.wikipedia.org/wiki/Bit" title="Bit">bit</a> code, meaning that it uses the bit patterns representable with seven binary digits (a range of 0 to 127 decimal) to represent character information.</p>
 125 </blockquote>
 126
 127 <p>Ergo US-ASCII is the characters whose ordinal values lie in the range of 0-127.</p>
 128
 129 <p>The CharacterSetValidation package installs a few Perl subroutines into the CQ schema that are supposed to check that the character data matches the code page of the database. However there doesn't seem to be any Perl code to check the difference between say a code page of US-ASCII or say Latin-1.</p>
 130
 131 <p>What it all comes down to eventual, as far as I can see, is a call to check_for_control_chars_in_line. The operative Perl code is:</p>
 132
 133 <div class="code"><pre>
 134 if ($line =~ /[^\t\n\r -\177]/) {
 135   return (" contains unsupported character(s):");
 136 }
 137 </pre></div>
 138
 139 <p>The above regex seems to be saying if the string $line is <b>not</b> a tab (\t), a linefeed (\n), a carriage return (\r) or in the range of space through \177 then an unsupported character has occurred. This does not include the entire US-ASCII range of 0-127. So if a client generated a yen character (¥) the above would not flag it as an unsupported character and yet (¥ - <span>ord</span> 165) is not a US-ASCII character.</p>
 140
 141 <p>Finally, since we only screened for US-ASCII in the 0-127 range, we have data that contains things like the US-ASCII Bel (ord 7) and the like that CharacterSetValidation is flagging as invalid.</p>
 142                 </div>
 143                 <nav class="page-navigation entry-navigation pagination content-nav">
 144                   <ul class="page-navigation-list">
 145
 146                     <li class="page-navigation-list-item page-navigation-prev"><a rel="prev" href="http://defaria.com/blogs/Status/2005/11/post-2.html" title="Stgloc/Code Page">Previous entry</a></li>
 147
 148
 149                     <li class="page-navigation-list-item page-navigation-next"><a rel="next" href="http://defaria.com/blogs/Status/2005/12/scripts.html" title="Cleardiffmrg/bin_merge problems">Next entry</a></li>
 150
 151                   </ul>
 152                 </nav>
 153                 <!--
 154 <aside id="zenback" class="zenback feedback">
 155   Please paste Zenback script code here.
 156 </aside>
 157 -->
 158
 159
 160               </article>
 161             </div>
 162             <aside class="widgets related" role="complementary">
 163               <nav class="widget-search widget">
 164   <div class="widget-content">
 165     <form method="get" id="search" action="http://defaria.com/mt/mt-search.cgi">
 166       <div>
 167         <input type="text" name="search" value="" placeholder="Search...">
 168
 169         <input type="hidden" name="IncludeBlogs" value="8">
 170
 171         <input type="hidden" name="limit" value="20">
 172         <button type="submit" name="button">
 173           <img alt="Search" src="/mt/mt-static/support/theme_static/rainier/img/search-icon.png">
 174         </button>
 175       </div>
 176     </form>
 177   </div>
 178 </nav>
 179 <nav class="widget-archive-category widget">
 180   <h3 class="widget-header">Categories</h3>
 181   <div class="widget-content">
 182
 183
 184     <ul class="widget-list">
 185
 186
 187       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/ameriquest/">Ameriquest (99)</a>
 188
 189
 190       </li>
 191
 192
 193
 194
 195       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/audience/">Audience (3)</a>
 196
 197
 198       </li>
 199
 200
 201
 202
 203       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/broadcom/">Broadcom (76)</a>
 204
 205
 206       </li>
 207
 208
 209
 210
 211       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/gpdb/">GPDB (35)</a>
 212
 213
 214       </li>
 215
 216
 217
 218
 219       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/general-dynamics/">General Dynamics (61)</a>
 220
 221
 222       </li>
 223
 224
 225
 226
 227       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/general-electric/">General Electric (13)</a>
 228
 229
 230       </li>
 231
 232
 233
 234
 235       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/hewlett-packard/">Hewlett Packard (13)</a>
 236
 237
 238       </li>
 239
 240
 241
 242
 243       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/lynuxworks/">LynuxWorks (162)</a>
 244
 245
 246       </li>
 247
 248
 249
 250
 251       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/pqa/">PQA (35)</a>
 252
 253
 254       </li>
 255
 256
 257
 258
 259       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/salira/">Salira (79)</a>
 260
 261
 262       </li>
 263
 264
 265
 266
 267       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/tellabs/">Tellabs (2)</a>
 268
 269
 270       </li>
 271
 272
 273
 274
 275       <li class="widget-list-item"><a href="http://defaria.com/blogs/Status/texas-instruments/">Texas Instruments (31)</a>
 276
 277
 278       </li>
 279
 280     </ul>
 281
 282
 283   </div>
 284 </nav>
 285
 286
 287 <nav class="widget-archive-dropdown widget">
 288   <h3 class="widget-header">Archives</h3>
 289   <div class="widget-content">
 290     <select>
 291       <option>Select a Month...</option>
 292
 293       <option value="http://defaria.com/blogs/Status/2016/02/">February 2016</option>
 294
 295
 296
 297       <option value="http://defaria.com/blogs/Status/2014/09/">September 2014</option>
 298
 299
 300
 301       <option value="http://defaria.com/blogs/Status/2014/04/">April 2014</option>
 302
 303
 304
 305       <option value="http://defaria.com/blogs/Status/2014/03/">March 2014</option>
 306
 307
 308
 309       <option value="http://defaria.com/blogs/Status/2013/02/">February 2013</option>
 310
 311
 312
 313       <option value="http://defaria.com/blogs/Status/2012/09/">September 2012</option>
 314
 315
 316
 317       <option value="http://defaria.com/blogs/Status/2012/08/">August 2012</option>
 318
 319
 320
 321       <option value="http://defaria.com/blogs/Status/2012/05/">May 2012</option>
 322
 323
 324
 325       <option value="http://defaria.com/blogs/Status/2012/04/">April 2012</option>
 326
 327
 328
 329       <option value="http://defaria.com/blogs/Status/2012/02/">February 2012</option>
 330
 331
 332
 333       <option value="http://defaria.com/blogs/Status/2012/01/">January 2012</option>
 334
 335
 336
 337       <option value="http://defaria.com/blogs/Status/2011/10/">October 2011</option>
 338
 339
 340
 341       <option value="http://defaria.com/blogs/Status/2011/07/">July 2011</option>
 342
 343
 344
 345       <option value="http://defaria.com/blogs/Status/2010/09/">September 2010</option>
 346
 347
 348
 349       <option value="http://defaria.com/blogs/Status/2010/08/">August 2010</option>
 350
 351
 352
 353       <option value="http://defaria.com/blogs/Status/2010/04/">April 2010</option>
 354
 355
 356
 357       <option value="http://defaria.com/blogs/Status/2010/03/">March 2010</option>
 358
 359
 360
 361       <option value="http://defaria.com/blogs/Status/2010/02/">February 2010</option>
 362
 363
 364
 365       <option value="http://defaria.com/blogs/Status/2009/05/">May 2009</option>
 366
 367
 368
 369       <option value="http://defaria.com/blogs/Status/2009/04/">April 2009</option>
 370
 371
 372
 373       <option value="http://defaria.com/blogs/Status/2008/07/">July 2008</option>
 374
 375
 376
 377       <option value="http://defaria.com/blogs/Status/2008/05/">May 2008</option>
 378
 379
 380
 381       <option value="http://defaria.com/blogs/Status/2008/04/">April 2008</option>
 382
 383
 384
 385       <option value="http://defaria.com/blogs/Status/2008/03/">March 2008</option>
 386
 387
 388
 389       <option value="http://defaria.com/blogs/Status/2008/02/">February 2008</option>
 390
 391
 392
 393       <option value="http://defaria.com/blogs/Status/2008/01/">January 2008</option>
 394
 395
 396
 397       <option value="http://defaria.com/blogs/Status/2007/12/">December 2007</option>
 398
 399
 400
 401       <option value="http://defaria.com/blogs/Status/2007/11/">November 2007</option>
 402
 403
 404
 405       <option value="http://defaria.com/blogs/Status/2007/10/">October 2007</option>
 406
 407
 408
 409       <option value="http://defaria.com/blogs/Status/2007/09/">September 2007</option>
 410
 411
 412
 413       <option value="http://defaria.com/blogs/Status/2007/08/">August 2007</option>
 414
 415
 416
 417       <option value="http://defaria.com/blogs/Status/2007/07/">July 2007</option>
 418
 419
 420
 421       <option value="http://defaria.com/blogs/Status/2007/06/">June 2007</option>
 422
 423
 424
 425       <option value="http://defaria.com/blogs/Status/2007/05/">May 2007</option>
 426
 427
 428
 429       <option value="http://defaria.com/blogs/Status/2007/04/">April 2007</option>
 430
 431
 432
 433       <option value="http://defaria.com/blogs/Status/2007/03/">March 2007</option>
 434
 435
 436
 437       <option value="http://defaria.com/blogs/Status/2007/01/">January 2007</option>
 438
 439
 440
 441       <option value="http://defaria.com/blogs/Status/2006/12/">December 2006</option>
 442
 443
 444
 445       <option value="http://defaria.com/blogs/Status/2006/11/">November 2006</option>
 446
 447
 448
 449       <option value="http://defaria.com/blogs/Status/2006/10/">October 2006</option>
 450
 451
 452
 453       <option value="http://defaria.com/blogs/Status/2006/09/">September 2006</option>
 454
 455
 456
 457       <option value="http://defaria.com/blogs/Status/2006/07/">July 2006</option>
 458
 459
 460
 461       <option value="http://defaria.com/blogs/Status/2006/06/">June 2006</option>
 462
 463
 464
 465       <option value="http://defaria.com/blogs/Status/2006/05/">May 2006</option>
 466
 467
 468
 469       <option value="http://defaria.com/blogs/Status/2006/04/">April 2006</option>
 470
 471
 472
 473       <option value="http://defaria.com/blogs/Status/2006/03/">March 2006</option>
 474
 475
 476
 477       <option value="http://defaria.com/blogs/Status/2006/02/">February 2006</option>
 478
 479
 480
 481       <option value="http://defaria.com/blogs/Status/2006/01/">January 2006</option>
 482
 483
 484
 485       <option value="http://defaria.com/blogs/Status/2005/12/">December 2005</option>
 486
 487
 488
 489       <option value="http://defaria.com/blogs/Status/2005/11/">November 2005</option>
 490
 491
 492
 493       <option value="http://defaria.com/blogs/Status/2005/10/">October 2005</option>
 494
 495
 496
 497       <option value="http://defaria.com/blogs/Status/2005/09/">September 2005</option>
 498
 499
 500
 501       <option value="http://defaria.com/blogs/Status/2005/08/">August 2005</option>
 502
 503
 504
 505       <option value="http://defaria.com/blogs/Status/2005/07/">July 2005</option>
 506
 507
 508
 509       <option value="http://defaria.com/blogs/Status/2005/06/">June 2005</option>
 510
 511
 512
 513       <option value="http://defaria.com/blogs/Status/2005/05/">May 2005</option>
 514
 515
 516
 517       <option value="http://defaria.com/blogs/Status/2005/04/">April 2005</option>
 518
 519
 520
 521       <option value="http://defaria.com/blogs/Status/2005/03/">March 2005</option>
 522
 523
 524
 525       <option value="http://defaria.com/blogs/Status/2005/02/">February 2005</option>
 526
 527
 528
 529       <option value="http://defaria.com/blogs/Status/2005/01/">January 2005</option>
 530
 531
 532
 533       <option value="http://defaria.com/blogs/Status/2004/12/">December 2004</option>
 534
 535
 536
 537       <option value="http://defaria.com/blogs/Status/2004/09/">September 2004</option>
 538
 539
 540
 541       <option value="http://defaria.com/blogs/Status/2004/08/">August 2004</option>
 542
 543
 544
 545       <option value="http://defaria.com/blogs/Status/2004/07/">July 2004</option>
 546
 547
 548
 549       <option value="http://defaria.com/blogs/Status/2004/06/">June 2004</option>
 550
 551
 552
 553       <option value="http://defaria.com/blogs/Status/2004/05/">May 2004</option>
 554
 555
 556
 557       <option value="http://defaria.com/blogs/Status/2004/04/">April 2004</option>
 558
 559
 560
 561       <option value="http://defaria.com/blogs/Status/2004/03/">March 2004</option>
 562
 563
 564
 565       <option value="http://defaria.com/blogs/Status/2004/02/">February 2004</option>
 566
 567
 568
 569       <option value="http://defaria.com/blogs/Status/2004/01/">January 2004</option>
 570
 571
 572
 573       <option value="http://defaria.com/blogs/Status/2003/12/">December 2003</option>
 574
 575
 576
 577       <option value="http://defaria.com/blogs/Status/2003/11/">November 2003</option>
 578
 579     </select>
 580   </div>
 581 </nav>
 582
 583
 584
 585 <div class="widget-syndication widget section">
 586   <div class="widget-content">
 587     <p><img src="http://defaria.com/mt/mt-static/images/status_icons/feed.gif" alt="Subscribe to feed" width="9" height="9" /> <a href="http://defaria.com/blogs/Status/atom.xml">Subscribe to this blog's feed</a></p>
 588
 589   </div>
 590 </div>
 591
 592             </aside>
 593           </div>
 594         </div>
 595         <footer id="footer" role="contentinfo">
 596           <div id="footer-inner">
 597             <div id="footer-content">
 598   <nav role="navigation">
 599           <ul>
 600             <li><a href="http://defaria.com/blogs/Status/">Home</a></li>
 601
 602
 603           </ul>
 604         </nav>
 605
 606   <p class="license">&copy; Copyright 2016.</p>
 607   <p class="poweredby">Powered by <a href="http://www.movabletype.org/">Movable Type</a></p>
 608 </div>
 609           </div>
 610         </footer>
 611       </div>
 612     </div>
 613     <script src="http://defaria.com/mt/mt-static/jquery/jquery.min.js"></script>
 614     <script src="http://defaria.com/blogs/Status/mt-theme-scale2.js"></script>
 615   </body>
 616 </html>