How to include variable input shape in convLSTM2D?

Question

I am classifying time series of 72x72 images in 4 filters (just like RGB). Things work well if all my samples have same time steps (or number of epochs). However, in reality I have different number of time steps per sample. (This is in astronomy, I cannot get data at my will.) But I get error when I try to include different time steps. Please look at the MWC.

N=2000
train_data_arr=np.random.rand(N, 11, 72, 72, 4)
train_label=np.repeat(np.random.randint(2,size=N)[:, np.newaxis], 11,axis=1)
Ntot,dum=train_label.shape;print(Ntot,dum)

#=== variable data shape ====
def select_random_time_epochs(data, labels, max_time_steps=11):
    variable_data = []
    variable_labels = []
    for d, l in zip(data, labels):
        #time_steps = 11
        time_steps = np.random.randint(1, max_time_steps + 1)
        variable_data.append(d[:time_steps])
        variable_labels.append(l[:time_steps])
    return variable_data, variable_labels
train_data_var, train_label_var = select_random_time_epochs(train_data_arr, train_label)

#==== data generator ======
def make_generator(data, labels):
    def generator():
        for d, l in zip(data, labels):
            yield d, l
    return generator

train_ds = tf.data.Dataset.from_generator(
    generator=make_generator(train_data_var, train_label_var),
    output_types=(tf.float32, tf.int32),
    output_shapes=(tf.TensorShape([None, 72, 72, 4]), tf.TensorShape([None]))
)
batch_size = 32
train_ds = train_ds.batch(batch_size)

#===model===
input_shape = (None, 72, 72, 4)  
model = tf.keras.Sequential()
model.add(Input(shape=input_shape))
model.add(ConvLSTM2D(32, (9, 9), activation='relu', padding='valid', return_sequences=True, data_format='channels_last'))
model.add(BatchNormalization())
model.add(TimeDistributed(MaxPooling2D((2, 2), data_format='channels_last')))
model.add(TimeDistributed(Flatten()))
model.add(TimeDistributed(Dense(64, activation='relu')))
model.add(TimeDistributed(Dense(1, activation='sigmoid')))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()

model.fit(train_ds, epochs=20)

If I keep the time steps fixed at 11 (in the fucntion select_random_time_epochs()), every thing works. But when I use variable number of time steps, I get the error:

Cannot batch tensors with different shapes in component 0. First element had shape [3,72,72,4] and element 3 had shape [6,72,72,4].

I understand that it cannot handle variable time steps within a batch of 32. Actually, when I set batch_size = 1, the above code works but this takes too much time and most-likely will never converge in the real use-case scenario.

So my questions are as follows.

1. Suppose I strictly do not want any padding whatsoever. Is there a faster way to implement model.fit() when the samples within a batch have different shapes? Otherwise, can I dynamically batch with the samples that have same time steps? Is this the only way? Will there be an issue while evaluating the test data which might not follow the same batch distribution?

2. Now coming to padding option: does tensorflow.keras.layers.Masking really take care of the 'bad images' that I put at the missing epochs? In other words, can Masking or something else can completely make the padding values irrelevant?

An important point: I have to use a generator to avoid loading data on GPU at once as my actual data is huge (literally astronomical). Also, the following threds discusses about the variable input for LSTM but my use case is slightly more complected.

https://stackoverflow.com/questions/63663399/how-to-handle-variable-length-data-for-lstm

https://stackoverflow.com/questions/38189070/how-do-i-create-a-variable-length-input-lstm-in-keras