It does take time for the refrigerant to cycle through the system cooling as it goes, yes. You may be a little low on refrigerant or perhaps the pollen filter is blocked, both of which will slow things down. Also if your TXV (Thermal Expansion Valve) is blocked or lazy this will slow things down further. However the work required to change the TXV is a lot so you'd have to consider the possible gains against doing the job.
When i was working on brand new cars 30 years ago, the usual figure was at least 10 minutes before the AC started "pulling down" - that's in the UK so in the warmth of Sri Lanka, i'd expect another 5-10 minutes.
The AC hasn't worked in my S Type since getting it so i can't offer a comparative figure but hoping to get it sorted fairly soon.