The time you are reffering to is the time you have to grab the attention of the visitor before they leave - approx 7 seconds :!:
Website speed is dependent on lots of different factors, here are just some of them...
Server Spec
Server Load
Server Connection(s)
Number of sites on the server (connected to server load).
General Network traffic.
Your ISP and it's routing ability.
Your connection speed / type
Your PC (or your visitors PC).
Type of website you have - static HTML pages load a lot quicker than dynamic pages (php for example) because dynamic pages require the accessing of a database to construct the page before it is served to visitors...
I have a speed test on my business hosting site which rates me approx 100 out of 250 in speed test results, now you may think that this is nothing to brag about - but I believe that in a shared hosting environment I am doing quite well and I dont mind publishing the results as they are 100% genuine... (I dont' claim figures that I can't backup :wink: )
Anyway - unless a site seriously times out then you are doing OK, you will only ever achieve blisteringly fast speeds with a dedicated server (and a good one at that)