6

在我的新闻页面项目中，我有一个数据库表news，其结构如下：

 - id: [integer] unique number identifying the news entry, e.g.: *1983*
 - title: [string] title of the text, e.g.: *New Life in America No Longer Means a New Name*
 - topic: [string] category which should be chosen by the classificator, e.g: *Sports*

此外，还有一个包含词频信息的表格贝叶斯：

 - word: [string] a word which the frequencies are given for, e.g.: *real estate*
 - topic: [string] same content as "topic" field above, e.h. *Economics*
 - count: [integer] number of occurrences of "word" in "topic" (incremented when new documents go to "topic"), e.g: *100*

现在我希望我的 PHP 脚本对所有新闻条目进行分类，并为它们分配几个可能的类别（主题）之一。

这是正确的实现吗？你能改进它吗？

<?php
include 'mysqlLogin.php';
$get1 = "SELECT id, title FROM ".$prefix."news WHERE topic = '' LIMIT 0, 150";
$get2 = mysql_abfrage($get1);
// pTOPICS BEGIN
$pTopics1 = "SELECT topic, SUM(count) AS count FROM ".$prefix."bayes WHERE topic != '' GROUP BY topic";
$pTopics2 = mysql_abfrage($pTopics1);
$pTopics = array();
while ($pTopics3 = mysql_fetch_assoc($pTopics2)) {
    $pTopics[$pTopics3['topic']] = $pTopics3['count'];
}
// pTOPICS END
// pWORDS BEGIN
$pWords1 = "SELECT word, topic, count FROM ".$prefix."bayes";
$pWords2 = mysql_abfrage($pWords1);
$pWords = array();
while ($pWords3 = mysql_fetch_assoc($pWords2)) {
    if (!isset($pWords[$pWords3['topic']])) {
        $pWords[$pWords3['topic']] = array();
    }
    $pWords[$pWords3['topic']][$pWords3['word']] = $pWords3['count'];
}
// pWORDS END
while ($get3 = mysql_fetch_assoc($get2)) {
    $pTextInTopics = array();
    $tokens = tokenizer($get3['title']);
    foreach ($pTopics as $topic=>$documentsInTopic) {
        if (!isset($pTextInTopics[$topic])) { $pTextInTopics[$topic] = 1; }
        foreach ($tokens as $token) {
            echo '....'.$token;
            if (isset($pWords[$topic][$token])) {
                $pTextInTopics[$topic] *= $pWords[$topic][$token]/array_sum($pWords[$topic]);
            }
        }
        $pTextInTopics[$topic] *= $pTopics[$topic]/array_sum($pTopics); // #documentsInTopic / #allDocuments
    }
    asort($pTextInTopics); // pick topic with lowest value
    if ($chosenTopic = each($pTextInTopics)) {
        echo '<p>The text belongs to topic '.$chosenTopic['key'].' with a likelihood of '.$chosenTopic['value'].'</p>';
    }
}
?>

培训是手动完成的，它不包含在此代码中。如果将文本“你可以通过出售房地产赚钱”分配给类别/主题“经济学”，那么所有单词（you,can,make,...）都将插入到表贝叶斯中，其中“经济学”为主题和 1作为标准计数。如果单词已经与相同的主题组合在一起，则计数会增加。

样本学习数据：

字数主题

卡钦斯基政治 1

索尼技术 1

银行经济学 1

电话技术1

索尼经济学 3

爱立信科技2

样本输出/结果：

文字标题：电话测试索尼爱立信阿斯彭-敏感温贝里

政治

....电话 ....测试 ....索尼 ....爱立信 ....阿斯彭 ....敏感 ....winberry

技术

....发现手机 ....测试 ....索尼发现 ....爱立信发现 ....aspen ....敏感 ....winberry

经济学

....电话 ....测试 ....发现索尼 ....爱立信 ....阿斯彭 ....敏感 ....温莓

结果：文本属于主题技术，可能性为 0.013888888888889

非常感谢您！

4

1 回答 1

7

看起来您的代码是正确的，但有一些简单的方法可以优化它。例如，您可以即时计算每个单词的 p(word|topic)，而您可以轻松地预先计算这些值。（我假设你想在这里对多个文档进行分类，如果你只做一个文档，我想这没关系，因为你不计算文档中没有的单词）

类似地，p(topic) 的计算可以移到循环之外。

最后，您无需对整个数组进行排序即可找到最大值。

都是小分！但这就是你要求的:)

我编写了一些未经测试的 PHP 代码，展示了如何在下面实现它：

<?php

// Get word counts from database
$nWordPerTopic = mystery_sql();

// Calculate p(word|topic) = nWord / sum(nWord for every word)
$nTopics = array();
$pWordPerTopic = array();
foreach($nWordPerTopic as $topic => $wordCounts)
{
    // Get total word count in topic
    $nTopic = array_sum($wordCounts);

    // Calculate p(word|topic)
    $pWordPerTopic[$topic] = array();
    foreach($wordCounts as $word => $count)
        $pWordPerTopic[$topic][$word] = $count / $nTopic;

    // Save $nTopic for next step
    $nTopics[$topic] = $nTopic;
}

// Calculate p(topic)
$nTotal = array_sum($nTopics);
$pTopics = array();
foreach($nTopics as $topic => $nTopic)
    $pTopics[$topic] = $nTopic / $nTotal;

// Classify
foreach($documents as $document)
{
    $title = $document['title'];
    $tokens = tokenizer($title);
    $pMax = -1;
    $selectedTopic = null;
    foreach($pTopics as $topic => $pTopic)
    {
        $p = $pTopic;
        foreach($tokens as $word)
        {
            if (!array_key_exists($word, $pWordPerTopic[$topic]))
                continue;
            $p *= $pWordPerTopic[$topic][$word];
        }

        if ($p > $pMax)
        {
            $selectedTopic = $topic;
            $pMax = $p;
        }
    }
} 
?>

至于数学...

您正在尝试最大化 p(topic|words)，所以找到

arg max p(topic|words)

（即 p(topic|words) 最高的论点主题）

贝叶斯定理说

                  p(topic)*p(words|topic)
p(topic|words) = -------------------------
                        p(words)

所以你正在寻找

         p(topic)*p(words|topic)
arg max -------------------------
               p(words)

由于文档的 p(words) 对于任何主题都是相同的，因此这与查找相同

arg max p(topic)*p(words|topic)

朴素贝叶斯假设（这使它成为朴素贝叶斯分类器）是

p(words|topic) = p(word1|topic) * p(word2|topic) * ...

所以使用这个，你需要找到

arg max p(topic) * p(word1|topic) * p(word2|topic) * ...

在哪里

p(topic) = number of words in topic / number of words in total

和

                   p(word, topic)                         1
p(word | topic) = ---------------- = p(word, topic) * ----------
                      p(topic)                         p(topic)

      number of times word occurs in topic     number of words in total
   = -------------------------------------- * --------------------------
            number of words in total           number of words in topic

      number of times word occurs in topic 
   = --------------------------------------
            number of words in topic
于 2010-09-02T12:18:01.537 回答