Skip to content

S
scan-optimization
提交 f7d96f43 authored 作者: qhz's avatar qhz
浏览文件
操作

【开发】增加第三版优化方案

上级 986fc827
显示空白字符变更
内嵌 并排
正在显示 包含 744 行增加 和 0 行删除
扫码二次统计方案( 第三版).md 0 → 100644
浏览文件 @ f7d96f43
# 背景
1. ydd_fw_scan_awards, ydd_fw_scans 数据已经达到千万级
2. ydd_fw_scan_awards, ydd_fw_scans 是个大宽表，字段极多，不同的业务使用不同字段。但对于特定业务来说有点累赘
3. ydd_fw_scan_awards, ydd_fw_scan 索引很多，但是 扫码统计很少能得上
# 修改点
1. 第3版, 去除 出货时间`out_day`
2. 使用 mongodb 怎么统计后数值存储服务
3. 使用 `day,dealer_id,store_id,product_id,province_id,city_id,district_id`　作为数组分组纬度
# 结果
从百万级别数据和千万级别的数据进行测试(亿级的没有测试)
千万级别的数据的最高查询时间高百万级别数据一个数量级。但是也没大于 `20s`, 具体细节可以查看`百万级数据测试(数量量:7800000, 容量大小: 867.67 MiB)` 和 `千万级数据测试(数量量:25440000, 容量大小:2.91G)`
`但是，所以数据都是虚构，存在数据倾斜问题。测试结果仅供参考`
# 业务员端、经销商端、门店端SQL
## 业务员端
### 数据概况
```
select
count(distinct(a.member_id)) as user_num,
SUM(if(a.is_point=1,a.point_num,0)) as grant_point_num,
SUM(if(a.is_bonus=1,a.bonus_num,0)) as grant_bonus_num,
SUM(if(a.is_gift=1,1,0)) as grant_gift_number,
SUM(if(a.is_many_awards=1 and a.is_award=1,1,0)) as is_many_awards_num,
SUM(if(s.is_first=1 and a.is_award=1,1,0)) as no_many_awards_num
from
ydd_fw_scan_awards as a
left join
ydd_fw_scans as s on a.scan_id = s.id
where
a.day BETWEEN 20240101 AND 20240530 and
s.product_id in (130433, 153727, 163915, 195850, 195852, 195860, 195861) and
s.dealer_id in (280, 259)
limit 1;
```
### 数据明细
```
select a.member_name,a.created_time,a.is_many_awards,a.activity_name,a.award_name,a.award_content,a.is_award,b.headimgurl,s.province,s.city,s.district,s.area
from
ydd_fw_scans as s
left join ydd_fw_scan_awards as a on s.id = a.scan_id
left join ydd_member as b on a.member_id = b.id
where
a.day BETWEEN 20240101 AND 20240630 and
s.product_id in (130433, 153727, 163915, 195850, 195852, 195860, 195861) and
s.dealer_id in (280, 259)
order by a.id desc
limit 0, 20
;
```
## 经销商端
### 数据概况
```
select
count(distinct(a.member_id)) as user_num,
SUM(if(a.is_point=1,a.point_num,0)) as grant_point_num,
SUM(if(a.is_bonus=1,a.bonus_num,0)) as grant_bonus_num,
SUM(if(a.is_gift=1,1,0)) as grant_gift_number,
SUM(if(a.is_many_awards=1 and a.is_award=1,1,0)) as is_many_awards_num,
SUM(if(s.is_first=1 and a.is_award=1,1,0)) as no_many_awards_num
from
ydd_fw_scan_awards as a
left join
ydd_fw_scans as s on a.scan_id = s.id
where
a.created_time BETWEEN 20240101 AND 20240530 and
s.province_id in (130433, 153727, 163915, 195850, 195852, 195860, 195861) and
s.city_id in (280, 259) and
s.district_id in (111, 222)
limit 1;
```
### 数据明细
```
select a.member_name,a.created_time,a.is_many_awards,a.activity_name,a.award_name,a.award_content,a.is_award,b.headimgurl,s.province,s.city,s.district,s.area
from
ydd_fw_scans as s
left join ydd_fw_scan_awards as a on s.id = a.scan_id
left join ydd_member as b on a.member_id = b.id
where
a.created_time BETWEEN 20240101 AND 20240530 and
s.province_id in (130433, 153727, 163915, 195850, 195852, 195860, 195861) and
s.city_id in (280, 259) and
s.district_id in (111, 222)
order by a.id desc
limit 0, 20
;
```
## 门店端
### 数据概况
```
select
count(distinct(a.member_id)) as user_num,SUM(if(a.is_point=1,point_num,0)) as grant_point_num,
SUM(if(a.is_bonus=1,bonus_num,0)) as grant_bonus_num,SUM(if(a.is_gift=1,1,0)) as grant_gift_number,
SUM(if(a.is_many_awards=1 and a.is_award=1,1,0)) as is_many_awards_num,
SUM(if(a.is_many_awards=0 and a.is_award=1,1,0)) as no_many_awards_num,
count(if(s.is_first=1,1,0)) as first_scan_num
from
ydd_fw_scan_awards as a
left join
ydd_fw_scans as s on a.scan_id = s.id
where
a.day BETWEEN 20240101 AND 20240530 and
s.day BETWEEN 20240101 AND 20240530 and
a.out_day BETWEEN 20240101 AND 20240530 and
s.out_day BETWEEN 20240101 AND 20240530 and
(s.product_id = 111,
s.product_sku_sn = 'xxxxx') or ((s.product_id in (1111,2222))
a.created_time BETWEEN 20240101 AND 20240530 and
s.province_id in (130433, 153727, 163915, 195850, 195852, 195860, 195861) and
s.city_id in (280, 259) and
s.district_id in (111, 222)
ｓ.store_id = x
s. is_first =1
limit 1;
```
### 数据明细
```
select
a.member_name,a.created_time,a.is_many_awards,a.activity_name,a.award_name,a.award_content,a.is_award,b.headimgurl,s.province,s.city,s.district,s.area
from
ydd_fw_scans as s
left join ydd_fw_scan_awards as a on s.id = a.scan_id
left join ydd_member as b on a.member_id = b.id
where
a.day BETWEEN 20240101 AND 20240530 and
s.day BETWEEN 20240101 AND 20240530 and
a.out_day BETWEEN 20240101 AND 20240530 and
s.out_day BETWEEN 20240101 AND 20240530 and
(s.product_id = 111,
s.product_sku_sn = 'xxxxx') or ((s.product_id in (1111,2222))
a.created_time BETWEEN 20240101 AND 20240530 and
s.province_id in (130433, 153727, 163915, 195850, 195852, 195860, 195861) and
s.city_id in (280, 259) and
s.district_id in (111, 222)
ｓ.store_id = x
s. is_first =1
order by a.id desc
limit 0, 20
;
```
# 二次统计方案
## 概述
- 把扫描数据拆分为两个表： 1. 扫描数据概述 2. 扫描数据明细
- 按日期：每天, 出货时间。进行数据写入。 写入的数据支持，覆写原有的数据
- 扫描数据概述的数据目的是进行每天统计写入
- 扫描数据明细为了缩减表字段和减少多表关联
## 数据写入SQL
### 扫描数据概况
以 scans_day(每天)、out_day, dealer_id、store_id, product_id 做唯一键。用于数据覆写
为了 把扫码时间和出货时间关联起来，必须要把out_day 作为分组的纬度
数据字段包括:
```
user_num,
grant_point_num,
grant_bonus_num,
grant_gift_number,
is_many_awards_num,
no_many_awards_num,
scans_day,
out_day,
province_id,
city_id,
district_id,
store_id,
dealer_id,
product_id
```
## 数量扫描量
用 `scans_day(每天), dealer_id、store_id, product_id` 与 `scans_day(每天)、out_day, dealer_id、store_id, product_id` 对比
```
select count(*) as k,day, dealer_id,store_id, product_id from ydd_fw_scans group by day, dealer_id, store_id, product_id having k > 100 order by k desc limit 10;
```
```
+-------+----------+-----------+----------+------------+
| k | day | dealer_id | store_id | product_id |
+-------+----------+-----------+----------+------------+
| 28856 | 20240212 | 0 | 0 | 195853 |
| 27812 | 20240212 | 0 | 0 | 0 |
| 27127 | 20240211 | 0 | 0 | 0 |
| 26914 | 20240211 | 0 | 0 | 195853 |
| 26675 | 20240215 | 0 | 0 | 195853 |
| 25655 | 20240213 | 0 | 0 | 195853 |
| 23430 | 20240214 | 0 | 0 | 195853 |
| 22789 | 20240213 | 0 | 0 | 0 |
| 22514 | 20240209 | 0 | 0 | 195853 |
| 21629 | 20240209 | 0 | 0 | 0 |
+-------+----------+-----------+----------+------------+
```
```
select count(*) from (
select
count(distinct(a.member_id)) as user_num,
SUM(if(a.is_point=1,a.point_num,0)) as grant_point_num,
SUM(if(a.is_bonus=1,a.bonus_num,0)) as grant_bonus_num,
SUM(if(a.is_gift=1,1,0)) as grant_gift_number,
SUM(if(a.is_many_awards=1 and a.is_award=1,1,0)) as is_many_awards_num,
SUM(if(s.is_first=1 and a.is_award=1,1,0)) as no_many_awards_num,
count(if(s.is_first=1,1,0)) as first_scan_num,
s.day,
s.dealer_id,
s.store_id,
s.product_id,
s.province_id,
s.city_id,
s.district_id
from
ydd_fw_scan_awards as a
left join
ydd_fw_scans as s on a.scan_id = s.id
where
a.day = 20240212
group by s.day,s.dealer_id, s.store_id, s.product_id, s.province_id, s.city_id, s.district_id
) as a;
+----------+
| count(*) |
+----------+
| 10001 |
+----------+
1 row in set (0.83 sec)
```
### 使用mongodb作为二次统计存储服务
### 百万级数据测试(数量量:7800000, 容量大小: 867.67 MiB)
### 测试语句1: 业务员端语句
```
// 搜索条件: day, dealer_id
db.scan_analy.aggregate([
{
$match: {
day: {$gt: 20240101, $lte: 20240601},
dealer_id: {$in:[20,28,12, 15]},
}
},
{
$group: {
_id:null,
user_num : {$sum: "$user_num"},
grant_point_num : {$sum: "$grant_point_num"},
grant_gift_number : {$sum: "$grant_gift_number"},
is_many_awards_num : {$sum: "$is_many_awards_num"},
no_many_awards_num : {$sum: "$no_many_awards_num"},
first_scan_num : {$sum: "$first_scan_num"},
}
},
{
$project: {
_id:0,
user_num: 1,
grant_point_num: 1,
grant_gift_number: 1,
is_many_awards_num: 1,
no_many_awards_num: 1,
first_scan_num: 1,
}
}
]).explain("executionStats");
```
运行结果
```
executionTimeMillis: 2888, // 查询执行耗时 2888 毫秒
totalKeysExamined: 400131, // 表示查询过程中检查了 400131 索引键
totalDocsExamined: 400000, // 查询检查了 400000 个文档
indexesUsed: [ //
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1',
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1'
],
```
```
// 搜索条件: day, dealer_id, product_id
db.scan_analy.aggregate([
{
$match: {
day: {$gt: 20240101, $lte: 20240601},
dealer_id: {$in:[20,28,12, 15]},
product_id: {$in: [100, 300, 400, 500, 800, 900, 44]}
}
},
{
$group: {
_id:null,
user_num : {$sum: "$user_num"},
grant_point_num : {$sum: "$grant_point_num"},
grant_gift_number : {$sum: "$grant_gift_number"},
is_many_awards_num : {$sum: "$is_many_awards_num"},
no_many_awards_num : {$sum: "$no_many_awards_num"},
first_scan_num : {$sum: "$first_scan_num"},
}
},
{
$project: {
_id:0,
user_num: 1,
grant_point_num: 1,
grant_gift_number: 1,
is_many_awards_num: 1,
no_many_awards_num: 1,
first_scan_num: 1,
}
}
]).explain("executionStats");
```
运行结果
```
executionTimeMillis: 36, // 查询执行耗时 36 毫秒
totalKeysExamined: 4731,, // 表示查询过程中检查了 4731, 索引键
totalDocsExamined: 4000, // 查询检查了 400000 个文档
indexesUsed: [ //
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1',
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1'
],
```
### 测试语句2: 经销商端语句
```
// 搜索条件: day, province_id,city_id,district_id
db.scan_analy.aggregate([
{
$match: {
day: {$gt: 20240101, $lte: 20240601},
province_id: {$gt: 80, $lte: 1000},
city_id: {$gt: 1000, $lte: 3000},
district_id: {$gt: 5, $lte: 7000}
}
},
{
$group: {
_id:null,
user_num : {$sum: "$user_num"},
grant_point_num : {$sum: "$grant_point_num"},
grant_gift_number : {$sum: "$grant_gift_number"},
is_many_awards_num : {$sum: "$is_many_awards_num"},
no_many_awards_num : {$sum: "$no_many_awards_num"},
first_scan_num : {$sum: "$first_scan_num"},
}
},
{
$project: {
_id:0,
user_num: 1,
grant_point_num: 1,
grant_gift_number: 1,
is_many_awards_num: 1,
no_many_awards_num: 1,
first_scan_num: 1,
}
}
]).explain("executionStats");
```
运行结果
```
executionTimeMillis: 3503, // 查询执行耗时 3503 毫秒
totalKeysExamined: 591788, // 表示查询过程中检查了 4731, 索引键
totalDocsExamined: 38159, // 查询检查了 400000 个文档
indexesUsed: [ //
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1',
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1'
],
```
### 测试语句3: 门店端语句
```
// 搜索条件: day, store_id, province_id,city_id,district_id
db.scan_analy.aggregate([
{
$match: {
day: {$gt: 20240101, $lte: 20240601},
store_id: 6,
province_id: {$gt: 80, $lte: 1000},
city_id: {$gt: 1000, $lte: 3000},
district_id: {$gt: 5, $lte: 7000}
}
},
{
$group: {
_id:null,
user_num : {$sum: "$user_num"},
grant_point_num : {$sum: "$grant_point_num"},
grant_gift_number : {$sum: "$grant_gift_number"},
is_many_awards_num : {$sum: "$is_many_awards_num"},
no_many_awards_num : {$sum: "$no_many_awards_num"},
first_scan_num : {$sum: "$first_scan_num"},
}
},
{
$project: {
_id:0,
user_num: 1,
grant_point_num: 1,
grant_gift_number: 1,
is_many_awards_num: 1,
no_many_awards_num: 1,
first_scan_num: 1,
}
}
]).explain("executionStats");
```
运行结果
```
executionTimeMillis: 345, // 查询执行耗时 345 毫秒
totalKeysExamined: 59142, // 表示查询过程中检查了 59142, 索引键
totalDocsExamined: 3763, // 查询检查了 3763 个文档
indexesUsed: [ //
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1',
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1'
],
```
```
// 搜索条件: day, store_id, product_id, province_id,city_id,district_id
db.scan_analy.aggregate([
{
$match: {
day: {$gt: 20240101, $lte: 20240601},
store_id: 6,
product_id: {$in: [100, 300, 400, 500, 800, 900, 44]},
province_id: {$gt: 80, $lte: 1000},
city_id: {$gt: 1000, $lte: 3000},
district_id: {$gt: 5, $lte: 7000}
}
},
{
$group: {
_id:null,
user_num : {$sum: "$user_num"},
grant_point_num : {$sum: "$grant_point_num"},
grant_gift_number : {$sum: "$grant_gift_number"},
is_many_awards_num : {$sum: "$is_many_awards_num"},
no_many_awards_num : {$sum: "$no_many_awards_num"},
first_scan_num : {$sum: "$first_scan_num"},
}
},
{
$project: {
_id:0,
user_num: 1,
grant_point_num: 1,
grant_gift_number: 1,
is_many_awards_num: 1,
no_many_awards_num: 1,
first_scan_num: 1,
}
}
]).explain("executionStats");
```
运行结果
```
executionTimeMillis: 20, // 查询执行耗时 20 毫秒
totalKeysExamined: 1163, // 表示查询过程中检查了 1163, 索引键
totalDocsExamined: 34, // 查询检查了 34 个文档
indexesUsed: [ //
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1',
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1'
],
```
### 千万级数据测试(数量量:25440000, 容量大小:2.91G)
### 测试语句1: 业务员端语句
```
// 搜索条件: day, dealer_id
db.scan_analy.aggregate([
{
$match: {
day: {$gt: 20240101, $lte: 20240601},
dealer_id: {$in:[20,28,12, 15,25]},
}
},
{
$group: {
_id:null,
user_num : {$sum: "$user_num"},
grant_point_num : {$sum: "$grant_point_num"},
grant_gift_number : {$sum: "$grant_gift_number"},
is_many_awards_num : {$sum: "$is_many_awards_num"},
no_many_awards_num : {$sum: "$no_many_awards_num"},
first_scan_num : {$sum: "$first_scan_num"},
}
},
{
$project: {
_id:0,
user_num: 1,
grant_point_num: 1,
grant_gift_number: 1,
is_many_awards_num: 1,
no_many_awards_num: 1,
first_scan_num: 1,
}
}
]).explain("executionStats");
```
运行结果
```
executionTimeMillis: 19024, // 查询执行耗时 19024 毫秒
totalKeysExamined: 2500126, // 表示查询过程中检查了 2500126 索引键
totalDocsExamined: 2500000, // 查询检查了 2500000 个文档
indexesUsed: [ //
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1',
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1'
],
```
```
// 搜索条件: day, dealer_id, product_id
db.scan_analy.aggregate([
{
$match: {
day: {$gt: 20240101, $lte: 20240601},
dealer_id: {$in:[20,28,12, 15,25]},
product_id: {$in: [200, 205, 300, 350, 101]}
}
},
{
$group: {
_id:null,
user_num : {$sum: "$user_num"},
grant_point_num : {$sum: "$grant_point_num"},
grant_gift_number : {$sum: "$grant_gift_number"},
is_many_awards_num : {$sum: "$is_many_awards_num"},
no_many_awards_num : {$sum: "$no_many_awards_num"},
first_scan_num : {$sum: "$first_scan_num"},
}
},
{
$project: {
_id:0,
user_num: 1,
grant_point_num: 1,
grant_gift_number: 1,
is_many_awards_num: 1,
no_many_awards_num: 1,
first_scan_num: 1,
}
}
]).explain("executionStats");
```
运行结果
```
executionTimeMillis: 312, // 查询执行耗时 312 毫秒
totalKeysExamined: 35126, // 表示查询过程中检查了 35126, 索引键
totalDocsExamined: 32500, // 查询检查了 32500 个文档
indexesUsed: [ //
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1',
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1'
],
```
### 测试语句2: 经销商端语句
```
// 搜索条件: day, province_id,city_id,district_id
db.scan_analy.aggregate([
{
$match: {
day: {$gt: 20240101, $lte: 20240601},
province_id: {$gt: 1300, $lte: 1500},
city_id: {$gt: 4700, $lte: 4800},
district_id: {$gt: 4500, $lte: 4800}
}
},
{
$group: {
_id:null,
user_num : {$sum: "$user_num"},
grant_point_num : {$sum: "$grant_point_num"},
grant_gift_number : {$sum: "$grant_gift_number"},
is_many_awards_num : {$sum: "$is_many_awards_num"},
no_many_awards_num : {$sum: "$no_many_awards_num"},
first_scan_num : {$sum: "$first_scan_num"},
}
},
{
$project: {
_id:0,
user_num: 1,
grant_point_num: 1,
grant_gift_number: 1,
is_many_awards_num: 1,
no_many_awards_num: 1,
first_scan_num: 1,
}
}
]).explain("executionStats");
```
运行结果
```
executionTimeMillis: 7743, // 查询执行耗时 7743 毫秒
totalKeysExamined: 1472989, // 表示查询过程中检查了 1472989 索引键
totalDocsExamined: 464, // 查询检查了 464 个文档
indexesUsed: [ //
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1',
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1'
],
```
### 测试语句3: 门店端语句
```
// 搜索条件: day, store_id, province_id,city_id,district_id
db.scan_analy.aggregate([
{
$match: {
day: {$gt: 20240101, $lte: 20240601},
store_id: 20,
province_id: {$gt: 1300, $lte: 1500},
city_id: {$gt: 4700, $lte: 4800},
district_id: {$gt: 4500, $lte: 4800}
}
},
{
$group: {
_id:null,
user_num : {$sum: "$user_num"},
grant_point_num : {$sum: "$grant_point_num"},
grant_gift_number : {$sum: "$grant_gift_number"},
is_many_awards_num : {$sum: "$is_many_awards_num"},
no_many_awards_num : {$sum: "$no_many_awards_num"},
first_scan_num : {$sum: "$first_scan_num"},
}
},
{
$project: {
_id:0,
user_num: 1,
grant_point_num: 1,
grant_gift_number: 1,
is_many_awards_num: 1,
no_many_awards_num: 1,
first_scan_num: 1,
}
}
]).explain("executionStats");
```
运行结果
```
executionTimeMillis: 242, // 查询执行耗时 242 毫秒
totalKeysExamined: 44488, // 表示查询过程中检查了 44488 索引键
totalDocsExamined: 19, // 查询检查了 19 个文档
indexesUsed: [ //
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1',
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1'
],
```
// 搜索条件: day, store_id, product_id, province_id,city_id,district_id
db.scan_analy.aggregate([
{
$match: {
day: {$gt: 20240101, $lte: 20240601},
product_id: {$in: [200, 205, 300, 350, 101]},
store_id: 12,
province_id: {$gt: 1300, $lte: 1500},
city_id: {$gt: 4700, $lte: 4800},
district_id: {$gt: 4500, $lte: 4800}
}
},
{
$group: {
_id:null,
user_num : {$sum: "$user_num"},
grant_point_num : {$sum: "$grant_point_num"},
grant_gift_number : {$sum: "$grant_gift_number"},
is_many_awards_num : {$sum: "$is_many_awards_num"},
no_many_awards_num : {$sum: "$no_many_awards_num"},
first_scan_num : {$sum: "$first_scan_num"},
}
},
{
$project: {
_id:0,
user_num: 1,
grant_point_num: 1,
grant_gift_number: 1,
is_many_awards_num: 1,
no_many_awards_num: 1,
first_scan_num: 1,
}
}
]).explain("executionStats");
```
运行结果
```
executionTimeMillis: 15, // 查询执行耗时 15 毫秒
totalKeysExamined: 960, // 表示查询过程中检查了 960, 索引键
totalDocsExamined: 34, // 查询检查了 34 个文档
indexesUsed: [ //
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1',
'day_1_dealer_id_1_store_id_1_product_id_1_province_id_1_city_id_1_district_id_1'
],
```
\ No newline at end of file
Markdown 格式
0% 或
您添加了 0 人 到此讨论。请谨慎行事。
请先完成此评论的编辑！
请 注册 或者 后发表评论