使用文本分析识别一段文本中的主要性别

构建性别词汇集

MALE = 'male'
FEMALE = 'female'
UNKNOWN = 'unknown' #表示这段句子既不关于男性也关于女性
BOTH = 'both' 

MALE_WORDS = set([
    'guy','spokesman','chairman',"men's",'men','him',"he's",'his',
    'boy','boyfriend','boyfriends','boys','brother','brothers','dad',
    'dads','dude','father','fathers','fiance','gentleman','gentlemen',
    'god','grandfather','grandpa','grandson','groom','he','himself',
    'husband','husbands','king','male','man','mr','nephew','nephews',
    'priest','prince','son','sons','uncle','uncles','waiter','widower',
    'widowers'
])

FEMALE_WORDS = set([
    'heroine','spokeswoman','chairwoman',"women's",'actress','women',
    "she's",'her','aunt','aunts','bride','daughter','daughters','female',
    'fiancee','girl','girlfriend','girlfriends','girls','goddess',
    'granddaughter','grandma','grandmother','herself','ladies','lady',
    'mom','moms','mother','mothers','mrs','ms','niece','nieces',
    'priestess','princess','queens','she','sister','sisters','waitress',
    'widow','widows','wife','wives','woman'
])
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

• 既然我们有了性别词汇集，我们需要一种方法来为句子分配性别，创建genderize函数，用于检查句子中出现MALE_WORDS列表和FEMAILE_WORDS列表中的单词数。如果一个句子只有MALE_WORDS，我们称为male句子，如果它只有FEMALE_WORDS,我们称为female句子。如果一个句子对男性和女性都有非零的计数值，我们称为both，如果它没有男性词汇也没有女性词汇，我们称为unknown:

构建分配性别的函数

def genderize(words):
    mwlen = len(MALE_WORDS.intersection(words))
    fwlen = len(FEMALE_WORDS.intersection(words))
    if mwlen > 0 and fwlen == 0:
        return MALE
    elif mwlen == 0 and fwlen > 0:
        return FEMALE
    elif mwlen > 0 and fwlen > 0:
        return BOTH
    else:
        return UNKNOWN
1
2
3
4
5
6
7
8
9
10
11

• 关于intersection（）方法，详见：Python 集合 intersection() 方法
• 我们需要一种方法，来计算文章中带有性别化的词汇和句子的频率，我们用Python的内置类collections.Counters类来进行计数。count_gender函数获取句子列表并应用genderize函数来评估性别化单词和性别化句子的总数。
• 每个句子的性别都被计算在内，句子中所有单词也被视为属于该性别：

统计性别化单词和性别化句子数量

from collections import Counter
def count_gender(sentences):
    sents = Counter()
    words = Counter()

    for sentence in sentences:
        gender = genderize(sentence) #判断这个句子类型，是关于男性还是女性，亦或其他类型
        sents[gender] += 1 #统计出现性别（男，女，或其他）的句子的数量
        words[gender] += len(sentence) #将组成性别的句子的单词数量都认定为这一性别的单词数
    return sents,words
1
2
3
4
5
6
7
8
9
10

• 关于Counter()方法，详见：Python collections.Counter用法
• 最后，为了使用性别计数器，我们还需要将文章的原始文本解析为句子和单词，为此我用NLTK库（我们会在本章后面进一步讨论）将段落分成句子。句子被断开后，可以将其词条化以判断每个单词和标点符号，再将词条化文本传递给性别计数器打印文档male,female,both或unknown百分比：

统计性别化句子及单词的百分比

import nltk
def parse_gender(text):
    sentences = [
        [word.lower() for word in nltk.word_tokenize(sentence)]
        for sentence in nltk.sent_tokenize(text)
    ]
    sents,words = count_gender(sentences)
    total = sum(words.values())

    for gender,count in words.items():
        pcent = (count/total) * 100
        nsents = sents[gender]

        print(
            "{0.3f}% {} ({} sentences)".format(pcent,gender,nsents)
              )
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

if __name__ == '__main__':
    with open('ballet.txt', 'r',encoding='utf8') as f:
        parse_gender(f.read())
1
2
3

39.269% unknown (48 sentences)
52.994% female (38 sentences)
4.393% both (2 sentences)
3.344% male (3 sentences)
1
2
3
4

• 在这里，评分函数在计算句子长度时考虑了所包含的单词数量。因此，尽管关于女性的句子总体上比较少，但超过50%的文章是关于女性的。对于这种技术进行扩展，就能分析男性句子和女性句子中单词的出现情况，看是否存在与男性或女性性别相关的其他常用词汇。

本文来源于《基于Python的智能文本分析》_Benjamin Bengfort等著，陈光译