In GSM, security is implemented in three entities: SIM card, GSM handset and Network. Subscriber identity module (SIM) contains: - IMSI - TMSI -PIN, -MSISDN -Authentication key Ki (64-bit) -Ciphering key (Kc) generating algorithm A8, and -Authentication algorithm A3. SIM is a single chip computer containing the operating system (OS), the file system, and applications. SIM is protected by a PIN and owned by an operator. SIM applications can be written with a SIM tool kit.

GSM handset contains ciphering algorithm A5.

Network uses algorithms

• A3 for Authentication,
• A5 for encryption: A5 is a stream cipher. It can be implemented very efficiently on hardware. Its design was never made public. A5 has several versions: A5/1 (most widely used today), A5/2 (weaker than A5/1; used in some countries), and A5/3 (newest version based on the Kasumi block cipher).
• A8 for ciphering the data; Ki and IMEI and IMSI of each subcriber are stored in the authentication center. Both A3 and A8 algorithms are implemented on the SIM. The operator can decide which algorithm is to be used. Implementation of an algorithm is independent of hardware manufacturers and network operators.

1. Authentication of GSM mobiles:

Authentication in the GSM system is achieved by the Base Station sending out a challenge to the mobile station. The MS uses a key stored on its SIM to send back a response that is then verified. This only authenticates the MS, not the user.

To request for a call or to receive a call, the MS has to get authenticated. The process is as follows:

• A unique subscriber authentication key is programmed on every SIM card. The authentication center (AuC) has a list which maps Ki number with the SIM card.It is a secure database.
• When a SIM card requests for a call, a 128 bit random number is instantaneously generated by the AuC and transmitted to the SIM card.
• The A3 algorithm which is programmed inside the SIM card processes the RAND number and Ki number and generates a 32 bit output called the Signed RESponse number (SRES).
• The same process is done on the AuC side.
• The SIM card transmits this SRES number to the AuC.
• The AuC compares the received SRES with the SRES that’s generated on the network side.
• The SIM is authenticated if and only if the two SRES are same.

The authentication centre contains a database of identification and authentication information for subscribers including IMSI, TMSI, location area identity (LAI), and authentication key (Ki). It is responsible for generating (RAND),response (RES), and ciphering key (Kc) which are stored in HLR / VLR for authentication and encryption processes. The distribution of security credentials and encryption algorithms provides additional security.

2. Encryption in GSM:

GSM uses information stored on the SIM card within the phone to provide encrypted communications and authentication. GSM encryption is only applied to communications between a mobile phone and the BS. The rest of the transmission over the normal fixed network or radio relay is unprotected, where it could easily be eavesdropped or modified. In some countries, the base station encryption facility is not activated at all, leaving the user completely unaware of the fact that the transmission is not secure. GSM encryption is achieved by the use of a shared secret key. If this key is compromised it will be possible for the transmission to be eavesdropped and for the phone to be cloned (i.e., the identity of the phone can be copied). A 64-bit key is divided to provide data confidentiality. It is not possible to encrypt all the data; for example, some of the routing information has to be sent in clear text. The detailed process of Encrypting the data is as shown in Figure 16.

• The AuC generates a random number (RAND ) of 128 bits and sends it to the MS.
• The RAND and the Ki number is processed by the A8 algorithm on both the sides.
• The A8 algorithm produces a 64 bit ciphering key (Kc). Ciphering means scrambling or randomizing the data.
• The A5 algorithm takes Kc key and data to be transmitted as input and and accordingly encrypts the data.
• A5 algorithm is different for each service provider and is highly secretive.

3. GSM Token-based challenge

The security-related information consisting of triplets of RAND, signature response (SRES), and Kc are stored in the VLR. When a VLR has used a token to authenticate an MS, it either discards the token or marks it used. When a VLR needs to use a token, it uses a set of tokens that is not marked as used in preference to a set that is marked used. When a VLR successfully requests a token from the HLR or an old VLR, it discards any tokens that are marked as used. When an HLR receives a request for tokens, it sends any sets that are not marked as used. Those sets shall then be deleted or marked as used. The system operator defines how many times a set may be reused before being discarded. When HLR has no tokens, it will query the authentication centre for additional tokens. The token-based challenge can be integrated into various call flows (e.g., registration, handoff). It is described separately here for clarity. Figure 17 shows the call flows of token-based challenges.

• The serving system sends a RAND to the MS.
• The MS computes the SRES using RAND and the authentication key (Ki) in the encryption algorithm.
• The MS transmits the SRES to the serving system.
• The MSC sends a message to the VLR requesting authentication.
• The VLR checks the SRES for validity.
• The VLR returns the status to the MSC.
• The MSC sends a message to the MS with a success or failure indication.

Both GSM and North American systems use the international mobile equipment identity (IMEI) stored in the equipment identity register (EIR) to check malfunctions and fraudulent equipment. The EIR contains a valid list (list of valid mobiles), a suspect list (list of mobiles under observation), and a fraudulent list (list of mobiles for which service is barred).