SQL Optimization Issues

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: sql 优化问题

| username: rw12306

Do you have any optimization suggestions for this SQL? It’s extremely slow right now.


EXPLAIN  analyze
select
	cf.MEDICAL_INSTITUT_CODE as orgId,
	cf.ORG_NAME as orgName,
	cf.PRESCRIPTION_NUMBER as pre_code,
	-- Prescription number
	cf.VISITING_SERIAL_NUMBER as serial_code,
	-- Outpatient number
	cf.PRESCRIPTION_PROPERTY as pre_type,
	-- Prescription type
	ifnull(cf.recipel_type, '1') as recipel_type,
	-- Prescription category
	cf.PRE_TIME as business_date,
	cf.FILE_NUMBER as patient_id,
	-- Patient ID
	br.patient_name,
	-- Patient name
	br.GENDER_CODE as sex,
	-- Gender code
	br.ID_TYPE as card_type,
	-- ID type
	br.ID_NUMBER as card_code,
	-- ID number
	br.BIRTH_DATE as birth_date,
	-- Birth date
	cf.PRE_DOCTOR_NAME as doc_name,
	-- Prescribing doctor
	cf.VISIT_DEP_NAME as dep_name,
	-- Prescribing department
	cf.DEPLOY_PHARMACIST_NAME as deploy_doc_name,
	-- Dispensing pharmacist
	cf.CHECK_PHARMACIST_NAME as check_doc_name,
	-- Verifying pharmacist
	cf.ISSUE_PHARMACIST_NAME as dispensing_doc_name,
	-- Issuing pharmacist
	cf.PRE_CHECK_DOC_CODE as examine_doc_name,
	-- Reviewing pharmacist
	cf.tcm_take_means,
	-- Traditional Chinese medicine taking method
	cf.tcm_decoction_method,
	-- Traditional Chinese medicine decoction method code
	antibiotic, 
	infusion_1,
	infusion_2,
	infusion_3,
	union_drug,
	base_drug,
	infusion,
	pre_amount,
	-- max(zd.diag_name) as diag_name, -- Diagnosis name
	zd.diag_name as diag_name,
	mx.drug_name
from
	yiliao.TB_CIS_OP_PRESCRIPTION cf use INDEX(PRIMARY)
inner join  
(
	select
		/*+ AGG_TO_COP() */
		mx.PRESCRIPTION_NUMBER,
		mx.MEDICAL_INSTITUT_CODE,
		GROUP_CONCAT(distinct mx.ANTIBIOTIC_SIGN order by mx.ANTIBIOTIC_SIGN) as antibiotic,
		-- Antibiotic sign
		max(case when mx.ANTIBIOTIC_SIGN = '1' then 1 else 0 end) as infusion_1,
		-- Non-restricted use antibiotics
		max(case when mx.ANTIBIOTIC_SIGN = '2' then 1 else 0 end) as infusion_2,
		-- Restricted use antibiotics
		max(case when mx.ANTIBIOTIC_SIGN = '3' then 1 else 0 end) as infusion_3,
		-- Special use antibiotics
		count(case when mx.ANTIBIOTIC_SIGN != '0' then mx.id else null end) as union_drug,
		-- Combined medication
		GROUP_CONCAT(distinct ifnull(mx.DRUG_BASE_ATTR_CODE, '1') order by mx.DRUG_BASE_ATTR_CODE) as base_drug,
		-- Drug base attribute
		max(case when mx.DRUG_USE_MEANS_CODE = '404' then 1 else 0 end) as infusion,
		-- Infusion prescription
		sum(mx.DRUG_AMOUNT) as pre_amount,
		-- Prescription amount	
		GROUP_CONCAT(distinct mx.ITEM_NAME) as drug_name
		-- Drug name collection
	from
		yiliao.TB_CIS_OP_PRESCRIPTION_DETAIL mx
	where
		mx.PRE_TIME >= '2021-01-01 00:00:00'
		and mx.PRE_TIME <'2021-01-07 00:00:00'
	group by
		mx.MEDICAL_INSTITUT_CODE,
		mx.PRESCRIPTION_NUMBER 
	) mx
on
	cf.PRESCRIPTION_NUMBER = mx.PRESCRIPTION_NUMBER
	and cf.MEDICAL_INSTITUT_CODE = mx.MEDICAL_INSTITUT_CODE
inner join 
(
	select /*+ AGG_TO_COP() */
		pat.FILE_NUMBER,
		pat.patient_name,
		-- Patient name
		pat.GENDER_CODE ,
		-- Gender code
		pat.ID_TYPE ,
		-- ID type
		pat.ID_NUMBER ,
		-- ID number
		date(pat.BIRTH_DATE) as BIRTH_DATE
		-- Birth date 
	from
		yiliao.TB_CIS_PATIENT_INFO pat
		##use index(PRIMARY)
	where
		FILE_NUMBER in (
		select
			FILE_NUMBER
		from
			yiliao.TB_CIS_OP_PRESCRIPTION_DETAIL mx
		where
			mx.PRE_TIME >= '2021-01-01 00:00:00'
			and mx.PRE_TIME <'2021-01-07 00:00:00'
)
)
br on
	cf.FILE_NUMBER = br.FILE_NUMBER
left join index_diagnosis zd USE index(index_diagnosis_serial_code_IDX) on
	cf.MEDICAL_INSTITUT_CODE = zd.orgid
	and cf.VISITING_SERIAL_NUMBER = zd.serial_code

| username: xfworld | Original post link

Can it be run separately? Or use temporary tables to solve part of the data pre-calculation problem?

| username: rw12306 | Original post link

Does TiDB support temporary tables? Right now, it can’t be split, the program is just like this.

| username: xfworld | Original post link

Not only does it support temporary tables, but it also supports CTE…

It only took 2.77 seconds in total, is that still considered slow?
Moreover, the data scan mostly used indexes, which is already very fast.

| username: rw12306 | Original post link

That only took 2.77 seconds. Querying a month of data already lags, and for a poor server, it takes tens of seconds.

| username: Billmay表妹 | Original post link

What version?

| username: xfworld | Original post link

After anonymizing the real environment, configuration, SQL, and plan, upload them.

You can also add the parts that are troubling you.

| username: rw12306 | Original post link

Temporary

6.1

| username: rw12306 | Original post link

This insert SQL is too slow. Are there any optimization solutions?

| username: buddyyuan | Original post link

PLAN REPLAYER, let’s extract the information and take a look. This SQL needs optimization.

| username: xfworld | Original post link

Is it just a simple insert into? Are there any other conditions?

| username: rw12306 | Original post link

It was inserted using the syntax “insert into (select aa from tt)”.

| username: xfworld | Original post link

Posting a picture always hides the problem points, can’t handle it… all the toothpaste has been squeezed out.

| username: rw12306 | Original post link

The zip file has been exported online
plan_replayer.zip (2.1 MB)

Executed SQL:

Insert Into index_prescription(`orgId`,`orgName`,`pre_code`,`serial_code`,`pre_type`,`recipel_type`,`business_date`,`patient_id`,`patient_name`,`sex`,`card_type`,`card_code`,`birth_date`,`doc_name`,`dep_name`,`deploy_doc_name`,`check_doc_name`,`dispensing_doc_name`,`examine_doc_name`,`tcm_take_means`,`tcm_decoction_method`,`antibiotic`,`infusion_1`,`infusion_2`,`infusion_3`,`union_drug`,`base_drug`,`infusion`,`pre_amount`,`diag_name`,`drug_name`) 
 select
	cf.MEDICAL_INSTITUT_CODE as orgId,
	cf.ORG_NAME as orgName,
	cf.PRESCRIPTION_NUMBER as pre_code,
	-- Prescription number
	cf.VISITING_SERIAL_NUMBER as serial_code,
	-- Outpatient number
	cf.PRESCRIPTION_PROPERTY as pre_type,
	-- Prescription type
	ifnull(cf.recipel_type, '1') as recipel_type,
	-- Prescription category
	cf.PRE_TIME as business_date,
	cf.FILE_NUMBER as patient_id,
	-- Patient ID
	br.patient_name,
	-- Patient name
	br.GENDER_CODE as sex,
	-- Gender code
	br.ID_TYPE as card_type,
	-- ID type
	br.ID_NUMBER as card_code,
	-- ID number
	br.BIRTH_DATE as birth_date,
	-- Birth date
	cf.PRE_DOCTOR_NAME as doc_name,
	-- Prescribing doctor
	cf.VISIT_DEP_NAME as dep_name,
	-- Prescribing department
	cf.DEPLOY_PHARMACIST_NAME as deploy_doc_name,
	-- Dispensing pharmacist
	cf.CHECK_PHARMACIST_NAME as check_doc_name,
	-- Checking pharmacist
	cf.ISSUE_PHARMACIST_NAME as dispensing_doc_name,
	-- Issuing pharmacist
	cf.PRE_CHECK_DOC_CODE as examine_doc_name,
	-- Reviewing pharmacist
	cf.tcm_take_means,
	-- TCM administration method
	cf.tcm_decoction_method,
	-- TCM decoction method code
	antibiotic, 
	infusion_1,
	infusion_2,
	infusion_3,
	union_drug,
	base_drug,
	infusion,
	pre_amount,
	-- max(zd.diag_name) as diag_name, -- Diagnosis name
	zd.diag_name as diag_name,
	mx.drug_name
from
	yiliao.TB_CIS_OP_PRESCRIPTION cf use INDEX(PRIMARY)
inner join  
(
	select
		/*+ AGG_TO_COP() */
		mx.PRESCRIPTION_NUMBER,
		mx.MEDICAL_INSTITUT_CODE,
		GROUP_CONCAT(distinct mx.ANTIBIOTIC_SIGN order by mx.ANTIBIOTIC_SIGN) as antibiotic,
		-- Antibiotic sign
		max(case when mx.ANTIBIOTIC_SIGN = '1' then 1 else 0 end) as infusion_1,
		-- Non-restricted antibiotic use
		max(case when mx.ANTIBIOTIC_SIGN = '2' then 1 else 0 end) as infusion_2,
		-- Restricted antibiotic use
		max(case when mx.ANTIBIOTIC_SIGN = '3' then 1 else 0 end) as infusion_3,
		-- Special antibiotic use
		count(case when mx.ANTIBIOTIC_SIGN != '0' then mx.id else null end) as union_drug,
		-- Combined medication
		GROUP_CONCAT(distinct ifnull(mx.DRUG_BASE_ATTR_CODE, '1') order by mx.DRUG_BASE_ATTR_CODE) as base_drug,
		-- Drug base attribute
		max(case when mx.DRUG_USE_MEANS_CODE = '404' then 1 else 0 end) as infusion,
		-- Infusion prescription
		sum(mx.DRUG_AMOUNT) as pre_amount,
		-- Prescription amount	
		GROUP_CONCAT(distinct mx.ITEM_NAME) as drug_name
		-- Drug name collection
	from
		yiliao.TB_CIS_OP_PRESCRIPTION_DETAIL mx
	where
		mx.PRE_TIME >= '2021-01-01 00:00:00'
		and mx.PRE_TIME < '2021-01-31 23:59:59'  
	group by
		mx.MEDICAL_INSTITUT_CODE,
		mx.PRESCRIPTION_NUMBER 
	) mx
on
	cf.PRESCRIPTION_NUMBER = mx.PRESCRIPTION_NUMBER
	and cf.MEDICAL_INSTITUT_CODE = mx.MEDICAL_INSTITUT_CODE
inner join 
(
	select
		pat.FILE_NUMBER,
		pat.patient_name,
		-- Patient name
		pat.GENDER_CODE ,
		-- Gender code
		pat.ID_TYPE ,
		-- ID type
		pat.ID_NUMBER ,
		-- ID number
		date(pat.BIRTH_DATE) as BIRTH_DATE
		-- Birth date 
	from
		yiliao.TB_CIS_PATIENT_INFO pat
		##use index(PRIMARY)
	where
		FILE_NUMBER in (
		select
			FILE_NUMBER
		from
			yiliao.TB_CIS_OP_PRESCRIPTION_DETAIL mx
		where
			mx.PRE_TIME >= '2021-01-01 00:00:00'
			and mx.PRE_TIME < '2021-01-31 23:59:59'  
)
)
br on
	cf.FILE_NUMBER = br.FILE_NUMBER
left join index_diagnosis zd USE index(index_diagnosis_serial_code_IDX) on
	cf.MEDICAL_INSTITUT_CODE = zd.orgid
	and cf.VISITING_SERIAL_NUMBER = zd.serial_code
| username: rw12306 | Original post link

What do you need to see here? I’ll get everything out for you.

| username: rw12306 | Original post link

This is the execution plan for the query statement.

| username: rw12306 | Original post link

I also see a lot of errors in the logs.

| username: rw12306 | Original post link

It’s already posted, please take a look.

| username: xfworld | Original post link

Here, a full index scan is performed, and the number of loops is very high with a relatively low hit rate. This is where the slowness comes from.

| username: xfworld | Original post link

I suggest you first check the health of these tables. If the health is not high enough, you can improve the health first and then see if there are any changes in the analysis results.