• The memory system is a hierarchy of storage devices with different capacities, costs, and access times.

• The idea centers on a fundamental property of computer programs known as locality. Programs with good locality tend to access the same set of data items over and over again, or they tend to access sets of nearby data items. Programs with good locality tend to access more data items from the upper levels of the memory hierarchy than programs with poor locality, and thus run faster. Figure 1 shows a typical memory hierarchy with relationship between the cost per bit, the access time and the hierarchy.

• Memory hierarchies work because well-written programs tend to access the storage at any particular level more frequently than they access the storage at the next lower level. So the storage at the next level can have a larger access time, and thus will be larger in size and cheaper per bit.

• The overall effect is a large pool of memory that costs as much as the cheap storage near the bottom of the hierarchy, but that serves data to programs at almost the rate of the fast storage near the top of the hierarchy.

1. Registers

• CPU registers are at the top most level of this hierarchy, they hold the most frequently used data. They are very limited in number and are the fastest.

• They are often used by the CPU and the ALU for performing arithmetic and logical operations, for temporary storage of data.

2. Cache

• The very next level consists of small, fast cache memories near the CPU. They act as
• staging areas for a subset of the data and instructions stored in the relatively slow main memory.
• There are often two or more levels of cache as well. The cache at the top most level after the registers is the primary cache. Others are secondary caches.
• Many a times there is cache present on board with the CPU along with other levels that are outside the chip.

3. Main Memory:

• The next level is the main memory, it stages data stored on large, slow disks often called hard disks. These hard disks are also called secondary memory, which are the last level in the hierarchy. The main memory is also called primary memory.

• The secondary memory often serves as staging areas for data stored on the disks or tapes of other machines connected by networks.

Performance Considerations: (Access Time vs cost vs size)

• A programmer needs to understand the memory hierarchy because it has a big impact on the performance of his applications.

• For example:

  If the data your program needs are stored in a CPU register, then they can be accessed in zero cycles during the execution of the instruction. If stored in a cache, 1 to 30 cycles. If stored in main memory, 50 to 200 cycles. And if stored in disk tens of millions of cycles. This is obviously delay being introduced.

• On the other hand, the cost per bit is the least for the secondary memory, which is more at the primary memory level. This increases till at the register level where it is the maximum.
• Similarly, memory sizes are smallest at the register level. This increases to a few KBs in the cache, few hundreds of MBs or even GBs nowadays in the main memory, to hundreds of GB’s in the secondary memory.
• Thus there is a tradeoff between cost, access time, size and all this needs to be taken into account when designing the memory hierarchy for a computer system.