What is the sign of a successful Lightning import?

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: lightning导入成功的标志是什么

| username: Soysauce520

【TiDB Usage Environment】Production Environment
【TiDB Version】v5.1.2
【Reproduction Path】lightning import
【Encountered Problem: No Successful mark after import completion, the last few lines of the log are as follows:
the whole procedure completed"] [takeTime=41m43.952921656s]
tidb lightning exit
service safe point keeper exited

The official documentation for the lightning section says that having this tidb lightning exit log means success, but this log also appears when manually terminated. How should I determine from the log?

| username: Fly-bird | Original post link

Is there a success flag?

| username: xingzhenxiang | Original post link

There will be a success message.

| username: xingzhenxiang | Original post link

I looked through my historical logs, please refer to the following:

| username: 像风一样的男子 | Original post link

Could you post more of your logs for us to take a look?

| username: TiDBer_小阿飞 | Original post link

Completed.

| username: hey-hoho | Original post link

The completion of the “restore table” phase in Lightning is just one stage and cannot be used as the final indicator of success.

| username: hey-hoho | Original post link

This is a very valuable question. Given that there are several statements in the official documentation, it is prudent to refer to the source code.

The core process of the Lightning service operation is as follows:
https://github.com/pingcap/tidb/blob/master/br/cmd/tidb-lightning/main.go#L76-L111

Focus on the parts related to log printing:

	finished := true
	if common.IsContextCanceledError(err) {
		err = nil
		finished = false
	}
	if err != nil {
		logger.Error("tidb lightning encountered error stack info", zap.Error(err))
		fmt.Fprintln(os.Stderr, "tidb lightning encountered error:", err)
	} else {
		logger.Info("tidb lightning exit", zap.Bool("finished", finished))
		exitMsg := "tidb lightning exit successfully"
		if (!finished) {
			exitMsg = "tidb lightning canceled"
		}
		fmt.Fprintln(os.Stdout, exitMsg)
	}

In other words, Lightning should print two logs when it runs normally:
Log file output: [tidb lightning exit] [finished=true]
Console output: tidb lightning exit successfully

Next, let’s see how the whole procedure completed comes about. The code:
https://github.com/pingcap/tidb/blob/master/br/pkg/lightning/importer/import.go#L526-L572
is the entire logic of Lightning import, divided into the following stages:

		rc.setGlobalVariables,
		rc.restoreSchema,
		rc.preCheckRequirements,
		rc.initCheckpoint,
		rc.importTables,
		rc.fullCompact,
		rc.cleanCheckpoints,

Before the process starts, a task called “the whole procedure” is initiated, and after the process ends, it is concluded with task.End(zap.ErrorLevel, err). The task has three final states:
https://github.com/pingcap/tidb/blob/master/br/pkg/lightning/log/log.go#L233-L256

	var verb string
	switch {
	case err == nil:
		level = task.level
		verb = " completed"
	case IsContextCanceledError(err):
		level = zap.DebugLevel
		verb = " canceled"
		extraFields = nil
	default:
		verb = " failed"
		extraFields = nil
	}

From this, it can be determined that when the log outputs “the whole procedure completed,” it indicates that the entire import is complete, and the time taken will be appended. Finally, back in the main function, the two logs mentioned earlier will be printed.

To summarize the success indicators of Lightning:
1. Log file prints [“the whole procedure completed”] [taketime=xxxx]
2. Log file prints [tidb lightning exit] [finished=true]
3. Console prints tidb lightning exit successfully

| username: Soysauce520 | Original post link

Very detailed, learned a lot.

| username: Soysauce520 | Original post link

The logs were manually typed, not suitable for production.

| username: Soysauce520 | Original post link

It should be.

| username: Soysauce520 | Original post link

There is no “success” in the logs.

| username: Soysauce520 | Original post link

Yes, you need to look at this, the exit line is not accurate.

| username: Soysauce520 | Original post link

Moderator, there’s also a warning line in the log: “warn, there are unfinished tables in the table meta table, cleanup skipped.” Is there any impact?

| username: xingzhenxiang | Original post link

The expert explained it clearly and concisely.

| username: hey-hoho | Original post link

This is a different question, I suggest starting a new thread for discussion :rofl:

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.