Python爬虫-爬取《星际穿越》短评并提取关键词

https://cdn.jsdelivr.net/gh/JMbaozi/Blogimg/Music/Hans%20Zimmer%20-%20Cornfield%20Chase.mp3

这次选择了我最喜欢的电影《星际穿越》的短评，共爬取了500条评论，并且用jieba分词提取了100个关键词。程序文件点我查看

100个关键词中有感情色彩的”情感”，”亲情”，”人性”，”煽情”权重值较高，也很好的反映了《星际迷航》剧情的走向和发展。”震撼”，”五星”，”神作”，”烧脑”等也说明了这部电影的深度和精彩。

剧情简介

近未来的地球黄沙遍野，小麦、秋葵等基础农作物相继因枯萎病灭绝，人类不再像从前那样仰望星空，放纵想象力和灵感的迸发，而是每日在沙尘暴的肆虐下倒数着所剩不多的光景。在家务农的前NASA宇航员库珀（马修·麦康纳 Matthew McConaughey 饰）接连在女儿墨菲（麦肯吉·弗依 Mackenzie Foy 饰）的书房发现奇怪的重力场现象，随即得知在某个未知区域内前NASA成员仍秘密进行一个拯救人类的计划。多年以前土星附近出现神秘虫洞，NASA借机将数名宇航员派遣到遥远的星系寻找适合居住的星球。在布兰德教授（迈克尔·凯恩 Michael Caine 饰）的劝说下，库珀忍痛告别了女儿，和其他三名专家教授女儿艾米莉亚·布兰德（安妮·海瑟薇 Anne Hathaway 饰）、罗米利（大卫·吉雅西 David Gyasi 饰）、多伊尔（韦斯·本特利 Wes Bentley 饰）搭乘宇宙飞船前往目前已知的最有希望的三颗星球考察。　　他们穿越遥远的星系银河，感受了一小时七年光阴的沧海桑田，窥见了未知星球和黑洞的壮伟与神秘。在浩瀚宇宙的绝望而孤独角落，总有一份超越了时空的笃定情怀将他们紧紧相连……

流程

爬取短评
保存评论
用jieba模块提取关键词
保存关键词及其权重值

游客访问豆瓣只能爬取11页评论，所以要先登入自己的账号，找到对应的Cookie作为标头，这样就可以爬取所有的短评了。(我把我的Cookie删掉了，你可以用你自己的Cookie来设置。)

1.爬取短评

def getComments(id,pageNum):
    movieComments = ""
    for i in range(pageNum):
        start = i*20
        url = "https://movie.douban.com/subject/"+str(id)+"/comments?start="+str(start)+"&limit=20&sort=new_score&status=P"
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36 Edg/81.0.416.45',
            'Cookie':''
        }
        print("正在爬取第%s页评论" % (i+1))
        r = requests.get(url,headers=headers)
        soup = BeautifulSoup(r.text,'lxml')
        commentsList = soup.find_all('span',class_ ='short')
        for comments in commentsList:
            movieComments += comments.text
            movieComments += '\n'
    return movieComments

2.保存评论

def saveComments(Comments):
    try:
        fileName = 'movieComments.txt'
        with open(fileName,'w',encoding='utf-8') as f:
            f.write(str(Comments))
        print('保存成功！')
    except:
        print('保存失败！')

3.用jieba模块提取关键词

keyWord = []
Weight = []
def getData():
    with open('movieComments.txt',encoding = 'utf-8') as f:
        data = f.read()
    for keyword, weight in extract_tags(data, topK=100, withWeight=True):
        keyWord.append(keyword)
        Weight.append(weight)

4.保存关键词及其权重值

def saveData(keyWord,Weight):
    with open('TF-IDFanalyse.txt','w') as f:
        for i in range(len(keyWord)):
            f.write(str(i+1)+'.'+str(keyWord[i])+' '+str(Weight[i]))
            f.write('\n')

完整代码

"""
Author:JMbaozi
Movie:星际穿越
MovieID:1889243
Total:500条
"""

import requests
import lxml
from bs4 import BeautifulSoup

def getComments(id,pageNum):
    movieComments = ""
    for i in range(pageNum):
        start = i*20
        url = "https://movie.douban.com/subject/"+str(id)+"/comments?start="+str(start)+"&limit=20&sort=new_score&status=P"
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36 Edg/81.0.416.45',
            'Cookie':''
        }
        print("正在爬取第%s页评论" % (i+1))
        r = requests.get(url,headers=headers)
        soup = BeautifulSoup(r.text,'lxml')
        commentsList = soup.find_all('span',class_ ='short')
        for comments in commentsList:
            movieComments += comments.text
            movieComments += '\n'
    return movieComments

def saveComments(Comments):
    try:
        fileName = 'movieComments.txt'
        with open(fileName,'w',encoding='utf-8') as f:
            f.write(str(Comments))
        print('保存成功！')
    except:
        print('保存失败！')

if __name__ == '__main__':
    id = '1889243'
    pageNum = 25
    Comments = getComments(id,pageNum)
    saveComments(Comments)

from jieba.analyse import *

keyWord = []
Weight = []
def getData():
    with open('movieComments.txt',encoding = 'utf-8') as f:
        data = f.read()
    for keyword, weight in extract_tags(data, topK=100, withWeight=True):
        keyWord.append(keyword)
        Weight.append(weight)

def saveData(keyWord,Weight):
    with open('TF-IDFanalyse.txt','w') as f:
        for i in range(len(keyWord)):
            f.write(str(i+1)+'.'+str(keyWord[i])+' '+str(Weight[i]))
            f.write('\n')

if __name__ == '__main__':
    getData()
    saveData(keyWord,Weight)
    print('保存成功！')

关键词

1.诺兰 0.19128255202316988
2.科幻 0.08669831467647365
3.电影 0.07004029275176318
4.穿越 0.043706565954566165
5.2001 0.040535205408894104
6.宇宙 0.0390774600021518
7.星际 0.03854840219222222
8.科幻片 0.03784300316051122
9.太空 0.03710094608307251
10.人类 0.034267650954529646
11.情感 0.033682035815611196
12.配乐 0.029996254863928012
13.亲情 0.028100484168683708
14.一部 0.025336134155516433
15.黑洞 0.0228315852980386
16.维度 0.022820988241827506
17.马修 0.02268071150495566
18.时间 0.021990215372328294
19.IMAX 0.021826649066327596
20.海瑟薇 0.021826649066327596
21.这部 0.021009323347174404
22.觉得 0.0209916098498522
23.叙事 0.02028916028157277
24.还是 0.02009185130318901
25.三体 0.019747920583820206
26.维空间 0.01946205433994088
27.人性 0.019195052934319248
28.震撼 0.018861075408388106
29.故事 0.018623987979694834
30.片子 0.018587782274193183
31.就是 0.01845701333304295
32.好看 0.017171096338749782
33.五星 0.016971453380785078
34.神作 0.01662982786005912
35.煽情 0.01644735800772909
36.地球 0.016405385658789774
37.视觉 0.016082692005066075
38.剧情 0.015890286414849592
39.不是 0.01581558745247783
40.最后 0.015788916147563902
41.时空 0.015733520464966094
42.布里克 0.015232252310259086
43.一个 0.014942671648402885
44.没有 0.014884828384911319
45.空间 0.014683676518727179
46.烧脑 0.01455109937755173
47.漫游 0.014546960040672925
48.电影院 0.014500702134194054
49.小时 0.01448711635342462
50.观影 0.014157740237741261
51.本片 0.014144234288097723
52.导演 0.013783215358711529
53.流浪 0.013716555891909232
54.影片 0.01330736993218049
55.安妮 0.01321389088855851
56.宏大 0.012998647802761259
57.其实 0.012928227988101199
58.观众 0.012789650481029386
59.剪辑 0.012779753415423404
60.看到 0.01268860125678143
61.致敬 0.012515144182016172
62.麦康纳 0.01247237089504434
63.好莱坞 0.012462577262208311
64.喜欢 0.012394775697748217
65.设定 0.011891436046434532
66.可以 0.011884859227459573
67.结尾 0.011831460504264474
68.伟大 0.011494687704687881
69.泪点 0.011433006653790646
70.盗梦 0.011433006653790646
71.作品 0.011248776462072684
72.铺垫 0.011231949525746828
73.情节 0.011034970375954618
74.自己 0.011026725613583723
75.理论 0.010972225292832552
76.真的 0.01077571993532777
77.世界 0.010632135572592594
78.我们 0.010614453730798123
79.很多 0.010559647606611895
80.因为 0.010559150068362024
81.内核 0.010531053899008868
82.拯救 0.010488712177217006
83.脑洞 0.01039364241253695
84.感觉 0.010300038608547208
85.这么 0.010241186809707007
86.地心引力 0.0101751461264128
87.什么 0.009923567535106937
88.非常 0.009804860819493999
89.简直 0.009784356439056686
90.科学 0.009698408562667363
91.如果 0.009654979314181883
92.女儿 0.00941427966544601
93.... 0.009354278171283256
94.二刷 0.009354278171283256
95.完全 0.00922557320115806
96.五维 0.009186249675812903
97.但是 0.009071528849504434
98.温情 0.009056552719912188
99.父女 0.009056552719912188
100.镜头 0.008980044681533647