How to compare the time cost of training the same model via different hardware architecture?

Question

For example, I plan to compare the performance of training the same model via:

1. 8 GPUs in one mode; vs
2. two nodes with 8 GPUs in each node (equals 16 GPUs).

How can I measure the time used the train the model in these 2 cases? I know in simple python code, I can use the cell magic %timeit to measure; but how to do this in large language model training? What things should I aware when comparing training time?

Thank you!

Answer 1

What things should I aware when comparing training time?

The only difference between LLM training code and regular Python code is the time it takes to ends. Therefore, it's preferable to time within the LLM training code, e.g. time to complete x batches.