If the sample size varies, doesn't this mean we have to keep calculating new Control Limits?

In theory, yes! But, for practical purposes, you need only re-calculate the Control Limits if the sample size varies by more than ±20%. It pays to have an ideal target size of sample and then plan your data collection to keep the actual sample sizes as close to this as possible. This avoids the need to re-calculate the Control Limits for each different sample which is, as you can imagine, a time-consuming business!