您好,我只想根据标题对电影进行聚类。我的函数对我的数据非常有效,但我有一个大问题,我的样本是 150.000 部大电影,实际上它非常慢需要 3 天才能对所有电影进行聚类
过程:
根据长度对电影标题进行排序
使用 countvectorizer 转换电影并计算每个电影的相似度(对于每个聚类电影,我每次都适合矢量化器并转换目标电影)
def product_similarity( clustered_movie, target_movie ):
'''
Calculates the title distance of 2 movies based on title
'''
# fitted vectorizer is a dictionary with fitted movies if wee dont fit to
# vectorizer the movie it fits and save it to dictionary
if clustered_movie in fitted_vectorizer:
vectorizer = fitted_vectorizer[clustered_movie]
a = vectorizer.transform([clustered_movie]).toarray()
b = vectorizer.transform( [target_movie] ).toarray()
similarity = cosine_similarity( a, b )
else:
clustered_movie = re.sub("[0-9]|[^\w']|[_]", " ",clustered_product )
vectorizer = CountVectorizer(stop_words=None)
vectorizer = vectorizer.fit([clustered_movie])
fitted_vectorizer[clustered_movie] = vectorizer
a = vectorizer.transform([clustered_movie]).toarray()
b = vectorizer.transform( [target_movie] ).toarray()
similarity = cosine_similarity( a, b )
return similarity[0][0]