nlp预处理常用函数

删除html标签

1
2
3

from bs4 import BeautifulSoup
a = BeautifulSoup()
a.get_text()

删除数字与标点符号

import re
letter_only = re.sub(&quot;[^a-zA-Z]&quot;,
                     &quot; &quot;,
                     example.get_text() )

转化大小写

1 2	lower_case = letter_only.lower() words = lower_case.split()

删除常用词

import nltk
nltk.download(&quot;stopwords&quot;)  # 按需下载字库
from nltk.corpus import stopwords
words = [w for w in words if not w in stopwords.words(&quot;english&quot;)]

转化为字符串形式

1	word = " ".join()