0

我正在使用 Powershell 7 处理 Wikipedia enwik9 1Gb UTF-8 文本文件。我没有使用 Unicode\UTF-8 的经验。我已经将偏移量和值捕获到一个字典中，当我使用下面的代码并增加 $i++ 时，它们似乎成对出现 2,4 和 6。

  1. $line.Length 对这个字符串有效吗？
  2. $i 是一个多字节字符，当它移动到下一次迭代时它仍然有效吗？
  3. 我怎么知道这段代码包含多少个“字符”？是 Substring($i,1) 还是 Substring($i,2) 或者 Substring($i,6)？
$text = (Get-Content 'enwik9.txt' -Raw)
$line = $text.Substring($i, 10000000)
for ($i = 0; $i -lt $line.Length; $i++) {
    $total_cnt++
    $s = $line.Substring($i, 1)
 
    $n = [int][CHAR]$s #I wanted [byte][char] here
    if ($n -ge 128) {
    # Now $n is not what I want because it is not ASCII and > 255 a Unicode\multibyte character
    }
}
4

1 回答 1

0

我能够根据此页面上的信息回答自己的问题并找到可行的解决方案： Ã © 和其他代码

clear-host
clear

write-host 'Loading enwik9.txt'
$text = (Get-Content 'enwik9.txt' -Raw)
write-host 'Load Complete - processing...'
 $line = $text.Substring($i,10000000)
  for($i=0;$i -lt $line.Length; $i++)
  {
  $total_cnt++

  $uni=''
  $s=$line.Substring($i,1)
  $n=[int][CHAR]$s

  if($n -ge 128)
  {
  # how many byte units in this Unicode?
  $ns=0
  $bin=0
  $n=$n-128 #reset the 8th contol bit
  $b7 = $n -band 64; if($b7 -eq 64){$ns=1;$n=$n-64} #remove the contorl bits
  $b6 = $n -band 32; if($b6 -eq 32){$ns-2;$n=$n-32}
  $b5 = $n -band 16; if($b5 -eq 16){$ns=3;$n=$n-16}
  $t=[convert]::ToString($n,16).PadLeft(2,'0')   #convert int to hex
  $bin= [convert]::tostring($n,2) 
  
  write-host 'Found a Unicode start byte $ns='$ns ' $n='$n
    for($c=1;$c -le $ns; $c++)
    {
    $i++; $total_cnt++;  #remember to increment the main loop index into #line
    $s=$line.Substring($i,1) #read the next string char
    $n=[int][CHAR]$s         #convert to int

    if($c -eq 1)
    {
    if(  (($n -band 128) -eq 128) -and (($n -band 64) -ne 0) ) 
    {
    write-host 'NOT A CONTINUE BIT $ns='$ns
    }

    $n=$n-128 #reset the 8th bit
    $b7 = $n -band 64; if($b7 -eq 64){$n=$n-64} #remove the contorl bits
   }

    $t=[convert]::ToString($n,16).PadLeft(2,'0') #convert int to hex
    $bin=$bin+ [convert]::tostring($n,2) 
    $number = [Convert]::ToInt32($bin, 2) #conver to int
    $hex = [convert]::ToString($number,16).PadLeft(4,'0')
    write-host '$s='$s ' $n='$n  ' $t='$t ' $bin='$bin ' $hex='$hex
    }

   $uc=''
   if($ns -eq 0){write-host 'SINGLE BYTE'; Read-Host 'ENTER';}
   ELSE{    $uni='\u'+$hex; $uc = [regex]::Unescape($uni) }


    write-host 'FINAL: Unicode is: '$uc
  read-host "press ENTER to find and process next unicode character"
  }
}
于 2020-10-04T21:02:26.410 回答