计算机毕业设计Hadoop+Spark+Hive知识图谱租房推荐系统 租房数据分析 租房爬虫 租房可视化 租房大数据 大数据毕业设计 大数据毕设 机器学习-Abstract

This design is a real estate data acquisition and visualization analysis application based on crawler technology. Firstly, the program collects all the housing data of real estate on the Internet through crawler, and cleans the collected data. These listings are roughly categorized to provide a summary of all the data. Through the above analysis, we can understand the basic characteristics of real estate on the market and the distribution of housing supply, which provides a reference for many home buyers to make purchase decisions.

The system is mainly composed of big data system, visual front-end system, Web background management system, rental recommendation system, rental small program /APP end. The large-screen statistical end is completed by Hadoop + Spark, data collection is developed by Java offline analysis end, web client end and background management using Springboot+ Mybatis framework. In the visualization stage, Echarts is used to provide interactive intuitive data visualization charts. The database used in this system is MySQL database, which is used to store a large number of rental information data sets obtained by crawler and the analysis results after data processing. Data analysis is completed through Spark parallel computing for data extraction, multidimensional analysis, query statistics and other operations. The development of a system integrating the analysis, recommendation, visualization and management of rental data based on big data.

Keywords: rental data analysis, big data development, Java development

目录

摘  要

Abstract

1 引 言

1.1大数据的发展

1.2 系统研究背景与意义

1.3 研究内容

2 系统分析

2.1 大数据分析较传统分析的优势

2.2 可行性分析

2.2.1 技术可行性

2.2.2 经济可行性

2.2.3 操作可行性

2.4 功能需求分析

3 开发技术介绍

3.1 硬件开发平台

3.1.1 计算机配置介绍

3.2 软件开发平台

3.2.1 WebMagic爬虫技术

3.2.2 MySQL数据库

3.2.3 Spark分析介绍

3.2.4 Spring Boot介绍

3.2.5 Vue开发

4 总体设计

4.1 大数据系统的设计

4.1.1 整体模块设计

4.1.2 数据采集功能设计

4.2 数据库设计

5 系统详细实现

5.1 数据采集功能实现

5.2 系统功能的实现

5.2.1 Spark框架进行数据分析

5.2.2 租房推荐页面的实现

5.2.3 web后端与可视化的实现

租房数据分析可视化流程

前台登录访问流程

系统管理界面

租房数据分析系统可视化界面

6 系统测试

6.1 系统测试工作概要

6.2 测试的意义

6.3 测试方法

7 总 结

致 谢

参考文献

核心算法代码分享如下：

from flask import Flask, request
import json
from flask_mysqldb import MySQL

# 创建应用对象
app = Flask(__name__)
app.config['MYSQL_HOST'] = 'bigdata'
app.config['MYSQL_USER'] = 'root'
app.config['MYSQL_PASSWORD'] = '123456'
app.config['MYSQL_DB'] = 'beike_hive'
mysql = MySQL(app)  # this is the instantiation


@app.route('/tables01')
def tables01():
    cur = mysql.connection.cursor()
    cur.execute('''SELECT * FROM table01''')
    #row_headers = [x[0] for x in cur.description]  # this will extract row headers
    row_headers = ['area','bads','goods']  # this will extract row headers
    rv = cur.fetchall()
    json_data = []
    #print(json_data)
    for result in rv:
        json_data.append(dict(zip(row_headers, result)))
    return json.dumps(json_data, ensure_ascii=False)

@app.route('/tables02')
def tables02():
    cur = mysql.connection.cursor()
    cur.execute('''SELECT * FROM table02''')
    #row_headers = [x[0] for x in cur.description]  # this will extract row headers
    row_headers = ['area','avg_pay']  # this will extract row headers
    rv = cur.fetchall()
    json_data = []
    #print(json_data)
    for result in rv:
        json_data.append(dict(zip(row_headers, result)))
    return json.dumps(json_data, ensure_ascii=False)

@app.route('/tables03')
def tables03():
    cur = mysql.connection.cursor()
    cur.execute('''SELECT * FROM table03 order by num desc''')
    #row_headers = [x[0] for x in cur.description]  # this will extract row headers
    row_headers = ['house_estate','num']  # this will extract row headers
    rv = cur.fetchall()
    json_data = []
    #print(json_data)
    for result in rv:
        json_data.append(dict(zip(row_headers, result)))
    return json.dumps(json_data, ensure_ascii=False)

@app.route('/tables04')
def tables04():
    cur = mysql.connection.cursor()
    cur.execute('''
    select * from (

SELECT ctime,num,CAST(replace(ctime,'小时前','') AS UNSIGNED) ctime2 FROM table04  where ctime  like '%小时前%' 
                                                      union all
SELECT ctime,num,CAST(replace(ctime,'天前','')*24 AS UNSIGNED) ctime2 FROM table04  where ctime  like '%天前%' 

)t order by t.ctime2 desc;
    ''')
    #row_headers = [x[0] for x in cur.description]  # this will extract row headers
    row_headers = ['ctime','num','ctime2']  # this will extract row headers
    rv = cur.fetchall()
    json_data = []
    #print(json_data)
    for result in rv:
        json_data.append(dict(zip(row_headers, result)))
    return json.dumps(json_data, ensure_ascii=False)

# @app.route("/getmapcountryshowdata")
# def getmapcountryshowdata():
#     filepath = r"D:\\hadoop_spark_hive_mooc2024\\server\\data\\maps\\china.json"
#     with open(filepath, "r", encoding='utf-8') as f:
#         data = json.load(f)
#         return json.dumps(data, ensure_ascii=False)


@app.route('/tables05')
def tables05():
    cur = mysql.connection.cursor()
    cur.execute('''SELECT * FROM table05''')
    #row_headers = [x[0] for x in cur.description]  # this will extract row headers
    row_headers = ['agent_name','hot']  # this will extract row headers
    rv = cur.fetchall()
    json_data = []
    #print(json_data)
    for result in rv:
        json_data.append(dict(zip(row_headers, result)))
    return json.dumps(json_data, ensure_ascii=False)

@app.route('/tables06')
def tables06():
    cur = mysql.connection.cursor()
    cur.execute('''SELECT * FROM table06''')
    #row_headers = [x[0] for x in cur.description]  # this will extract row headers
    row_headers = ['house_type','num']  # this will extract row headers
    rv = cur.fetchall()
    json_data = []
    #print(json_data)
    for result in rv:
        json_data.append(dict(zip(row_headers, result)))
    return json.dumps(json_data, ensure_ascii=False)

@app.route('/tables07')
def tables07():
    cur = mysql.connection.cursor()
    cur.execute('''SELECT * FROM table07''')
    #row_headers = [x[0] for x in cur.description]  # this will extract row headers
    row_headers = ['house_decora','num']  # this will extract row headers
    rv = cur.fetchall()
    json_data = []
    #print(json_data)
    for result in rv:
        json_data.append(dict(zip(row_headers, result)))
    return json.dumps(json_data, ensure_ascii=False)

@app.route('/tables08')
def tables08():
    cur = mysql.connection.cursor()
    cur.execute('''SELECT * FROM table08''')
    #row_headers = [x[0] for x in cur.description]  # this will extract row headers
    row_headers = ['house_pay_way','num']  # this will extract row headers
    rv = cur.fetchall()
    json_data = []
    #print(json_data)
    for result in rv:
        json_data.append(dict(zip(row_headers, result)))
    return json.dumps(json_data, ensure_ascii=False)

@app.route('/tables09')
def tables09():
    cur = mysql.connection.cursor()
    #cur.execute('''SELECT SUBSTRING(address) address,num FROM table09''')
    cur.execute('''SELECT SUBSTRING(address,-5) address,num FROM table09''')
    #row_headers = [x[0] for x in cur.description]  # this will extract row headers
    row_headers = ['address','num']  # this will extract row headers
    rv = cur.fetchall()
    json_data = []
    #print(json_data)
    for result in rv:
        json_data.append(dict(zip(row_headers, result)))
    return json.dumps(json_data, ensure_ascii=False)


if __name__ == "__main__":
    app.run(debug=False)