Back to Tylogix Home Page

Tunneling Into the Internet: Understanding VPN's


By Thibault Dambrine



In the traditional world of hand-to hand, face-to-face world of business transactions, companies rely on a number of accepted guarantees, formal and informal, for each transaction. As the world of technology is allowing us to do more business, more remotely, tools such as the telephone, fax machines and now the Internet have gained acceptance to the point where they are now basic necessities.

To resume the basic pillars of these traditional transactions, you could say that as a rule, commercial transactions rely on the following characteristics of trust:

These are in fact the principles that have spurred the creation of VPN protocols. Without these guarantees, doing business on the Internet would be far more risky and far less attractive to most companies. Without VPN technology, IP data packets travel through the Internet "in the clear" - meaning readable by anyone on their path. Exposures such as sniffing spoofing, and session hijacking are an obvious concern. There is also no form of non-repudiation for contractual transactions, an obvious handicap for doing business.

IPSec and PPTP are currently the most popular options for VPN technology. These two VPN solutions differ vastly in their method and in their implementation. Both however aim to grant their users the advantage of using the public Internet for private purposes.

This article will cover the following topics:




TCP/IP Brief

The function or purpose of Internet Protocol (IP) is to move data in the form of small packages or datagrams through an interconnected set of networks. To achieve this goal, IP passes these datagrams from one Internet module to another until the destination is reached. Each datagram is similar to a hand-written letter you would send through the mail. It has an envelope, with address information on it, and a content, where the data you are aiming to transmit is stored.

The Internet Protocol Operation: TCP/IP

The most well known IP protocol is TCP/IP. TCP stands for Transport Control Protocol. Whereas the IP protocol deals only with packets, TCP enables two hosts to establish a connection and exchange streams of data. With a system of confirmations and verifications, TCP uses the IP service to deliver data and guarantee that packets will be delivered in the same order in which they were sent. What this means is that if a packet was lot for example, TCP will use IP to re-transmit it.

Transmission Control Protocol (TCP) is a means for building a reliable communications stream on top of the unreliable packet Internet Protocol (IP). TCP is the protocol that supports nearly all Internet applications. The combination of TCP and IP is referred to as TCP/IP. Many people view TCP/IP incorrectly, as a single protocol.

The basic method of operation involves


Anatomy of a TCP/IP Datagram

| Source Port | Destination Port |
| Sequence Number |
| acknowledgement Number |
| Header Length |URG |ACK |PSH |RST |SYN |FIN | Window Size |
| Checksum | Urgent Pointer |
| Options |







(IP Header)

| data |
\ data \
\ data \
| data |



Data Portion




IPV6: The new Internet Protocol

The IETF (Internet Engineering Task Force) is the body that sets the standards for the Internet. It is an open international community of network designers, operators, vendors, and researchers concerned with the evolution of the Internet architecture and the smooth operation of the Internet.

Devised by the IETF, the IPv6 Internet protocol aims to improve the current IPV4 Internet technology on several fronts. Here is a short list of these improvements. Note that some of the features, such as the Security features, can and are already implementable using Ipv4, the current version of the Internet.


IPV6 has probably not made front-page news in your local paper but the Internet Assigned Numbers Authority - IANA - ( is already granting IPV6 addresses since September 1999.



VPN concepts

The purpose of a Virtual Private Network, or VPN is to create a "tunnel" that would ensure a secure and private data link between two points using a public network such as (most commonly now) the Public Internet.

Typically, the two endpoints of a VPN will exchange data transparently, unaware of the encryption, compression and hops that separate them. The "Virtual" part of VPN identifies the fact that you are building a private link over a public network, rather than actually purchasing a hard-wired link over a leased line. Using a VPN involves encrypting data before sending it through the public network and decrypting it at the receiving end. An additional level of security involves encrypting not only the data but also the originating and receiving network addresses. This helps in not revealing information about your internal network addresses.

A virtual private network can be contrasted with a system of owned or leased lines that can only be used by one company. The idea of the VPN is to give the company the same capabilities with greater flexibility at much lower cost by using the shared public infrastructure rather than a private one. Phone companies have provided secure shared resources for voice messages. A VPNmakes it possible to share the public resource of the Internet while keeping the data private and secure.

How would a VPN be useful to your company?


IPSec Functionality: Three Parties, One Channel, And Two Modes

IPSec supports two encryption modes: Transport and Tunnel. Transport mode encrypts only the data portion (payload) of each packet, but leaves the header (containing IP addreses and other info) untouched. The more secure Tunnel mode encrypts both the header and the payload. On the receiving side, an IPSec-compliant device decrypts each packet. Only one stream of packets (channel) is necessary to support an IPSec data exchange.


Originally engineered and proposed by Cisco Systems, as part of the IETF, the IPSec aims to addresses three basic security questions:

  1. Integrity: ensuring that the packets of data transmitted from point A to point B have not been altered on their way during the transmission.
  2. Confidentiality: ensuring that the transmitted data cannot be observed or read on its way to destination.
  3. Authentication: ensuring that the person (machine) you are talking to is really that person (machine).
  4. Replay protection ensures that an attacker cannot intercept a datagram and play it back at some later time without being detected.
  5. Automated management of cryptographic keys and security associations assures that a company's VPN policy can be conveniently and accurately implemented throughout the extended network with little or no manual configuration


IPSec uses the following methods to address the original three security questions:

Transport Mode IPSec

Tunnel Mode IPSec



TCP/IP, UDP or other IP Protocol

TCP/IP, UDP or other IP Protocol

IPSec Security Layer

Outer IP Address (gateway or firewall for example)

IP Address

IPSec Security Layer

Payload (data)

Inner IP Address (the ultimate destination of the packet beyond the gateway or firewall)

Physical Layer (hardware)

Payload (data)


Physical layer (hardware)


  1. Authentication Header (AH) Protocol: The AH is used to ensure the integrity of the IP packet. It does not however provide data confidentiality. In Transport mode, the AH is inserted after the IP header and before the upper layer protocol (such as TCP or UDP). In Tunnel mode, AH verifies the integrity of the entire IP packet, including the entire inner IP header.
  2. Encapsulating Security Payload (ESP) Protocol: ESP provides encryption, integrity and authentication. In Transport mode, it provides protection for upper layer protocols (TCP or UDP for example) but not the IP header. In Tunnel mode, the "inner IP header carries the ultimate source and destination addresses, while an "outer" IP header may contain a distinct IP address (one of a firewall or security gateway). Still in Tunnel mode, ESP protects the entire inner IP packet, including the entire inner IP header.
  3. IKE - The Internet Key Exchange - is at the base of the "knowing who you are talking to is really that person". Here is how it works: IKE uses Security Associations (SA) with a trusted entity.

Before getting into the meat of IKE, here is a quick recap on asymmetric key encryption. Here is the first paragraph of the definition: A cryptographic system that uses two keys -- a public key known to everyone and a private or secret key known only to the recipient of the message. When John wants to send a secure message to Jane, he uses Jane's public key to encrypt the message. Jane then uses her private key to decrypt it.

An example of a key exchange can be as follows: Each of the users aiming to communicate exchange public key values. Each user then mathematically combine the other's public key along with their own secret information to compute a secret value. This secret value can be used a s a session key or as an encryption key for encrypting a randomly generated session key. This method generates a session key based on public and secret information held by both users. The benefit of this method is that the key used for encrypting messages is based on information held by both users. Also, the independence of keys from one key exchange to another provides additional secrecy. There are several key exchange methods, this is one of the more representative techniques used.

Certificate Authority (CA) - This is used to ensure two parties have strong assurance that who they are communicating with really is who they claim to be. Certificate Authority is another name for "third-party verification". In this case, it is a trusted third party that can vouch for the connecting parties.

To verify a host taking part in an IPSec exchange, three elements are necessary:

The host identity (the IP Address)

The host's public key (the host certificate)

The CA's public key (the CA certificate)

An individual wishing to send an encrypted message applies for a digital certificate from a Certificate Authority (CA).The CA issues an encrypted digital certificate containing the applicant's public key and a variety of other identification information. The CA makes its own public key readily available through print publicity or on the Internet.

The recipient of an encrypted message uses the CA's public key to decode the digital certificate attached to the message, verifies it as issued by the CA and then obtains the sender's public key and identification information held within the certificate. With this information, the recipient can send an encrypted reply.

The business of providing CA's is often referred to as PKI, or Public Key Infrastructure. By necessity, PKI providers must be managed by entities with a high level of trust in the community. By design, in a security world, nobody should be trusted unconditionally. This principle applies equally (perhaps especially) to CA's. Thus, CA's are themselves certified by other CA's. Examples of CA's include VeriSign, The CommerceNet CA, and the U.S. Postal Service.

The Cisco Secure VPN Client provides Microsoft Windows 95/98 and NT 4.0 users with complete implementation of IPSec standards, including support for DES and Triple DES encryption and authentication with digital certificates, one-time password tokens, and pre-shared keys.


On the Positive side:

At time of writing, IPSec is the current "best practice" for VPN standards. No other standard fills the mission of security and authentication as well in a VPN solution.

On the Negative side:

Many experts question the usefulness of the Transport mode. First, it is less secure than the Tunnel mode, second, having two modes instead of one tends to add complexity.

Too many complexities means more possible breaking points.

The documentation is hard to understand


PPTP: Two Parties, a Control Channel and a Session channel

Short for Point-to-Point Tunneling Protocol, PPTP is a technology for creating Virtual Private Networks (VPN's) , developed jointly by Microsoft Corporation, U.S. Robotics, and several remote access vendor companies, known collectively as the PPTP Forum.

Although PPTP has been submitted to the IETF for standardization, it is currently available only on networks served by a Windows 2000 server and Linux.

Microsoft PPTP [Mic96a, Mic96b] is sold as part of the Windows NT Server. It can also be downloaded without charge from the Microsoft Windows website. PPTP can be enabled using the Windows Network Control Panel and the Registry Editor. This implementation of PPTP is used extensively in commercial VPN products precisely because of its availability in NT.

PPTP uses a number of terms that may not be familiar. Before starting, here is a list of these terms:

The references in square brackets can be looked up conveniently on the Internet.

PPP: Point to Point Protocol [Sim94]

PPTP: Point to Point Tunneling Protocol [Mic96a, Mic96b]]

PNS: PPTP Network Server

PAC: PPTP Access Concentrator

GRE: Generic Routing Encapsulation [RFC1701]

(The Internet Drafts numbers in [square brackets] can be looked up on the Internet)


In PPTP, there are two parallel components:

  1. A control connection between each PAC-PNS pair operating over TCP.
  2. An IP tunnel operating between the same PAC-PNS pair that is used to transport GRE encapsulated PPP packets for user sessions between the pair.

Here is how PPTP works:

Control and Session Channel Functionality:

Following the establishment of the TCP connection, the PNS and PAC first establish the control connection or control channel that will rule the exchange.

Once the control channel is established between the two parties, the PAC or PNS may initiate sessions by requesting outbound calls or responding to inbound requests. Individual sessions may be released by either of the PAC or PNS, using the control channel.

The control channel is maintained by a keep-alive echo signal. This ensures that a connectivity failure is promptly detected.

The Tunnel Protocol:

PPTP requires the establishment of a tunnel for each communicating PNS-PAC pair. This tunnel is used to carry all user session PPP packets for sessions involving a given PNS-PAC pair. PPP (Point to Point Protocol) is a protocol that allows a computer connecting via modem to act like a computer connected directly to the network. This enables a graphical user interface for Web access, FTP, e-mail, and Telnet.

A key, which is present in the GRE header, indicates which session a particular PPP packet belongs to. In this manner, PPP packets are multiplexed and de-multiplexed over a single tunnel between a given PNS-PAC pair. The value to use in the key field is established by the call establishment procedure, which takes place at the control connection time.


Putting it all together:

The control channel is a standard TCP connection to port 1723 on the server. The data channel carrying the private network data uses GRE, a generic encapsulation protocol The transparent transmission over the data channel is achieved by negotiating a standard PPP connection over it, just as if it were a dialup connection directly from the client to the server. The options negotiated over the tunnel by PPP control whether the data is compressed and/or encrypted. Thus, PPP itself has nothing to do with encryption..


PPTP uses CHAP (Challenge Handshake Authentication Protocol), an authentication protocol where the two peers know a common secret hash.

CHAP is used to periodically verify the identity of the peer using a 3-way handshake. This is done upon initial link establishment, and MAY be repeated anytime after the link has been established.

  1. 1 After the Link Establishment phase is complete, the authenticator sends a "challenge" message to the peer.
  2. The peer responds with a value calculated using a "one-way hash" function.
  3. The authenticator checks the response against its own calculation of the expected hash value. If the values match, the authentication is acknowledged; otherwise the connection SHOULD be terminated.
  4. At random intervals, the authenticator sends a new challenge to the peer, and repeats steps 1 to 3.


On the Positive side:

On the Negative side:


Pros and Cons

Comparing the characteristics of PPTP with those of IPSec as outlined above, one could easily describe one as light duty (PPTP) and the other as industrial strength (IPSec).

However, this is not the whole story. Looking closer at these two VPN solutions, one could make the following statements:

Both are better than transmitting unencrypted information over the Internet. While PPTP is weaker than IPSec, it is much easier and much cheaper to implement. It also has a huge critical mass in the market by virtue of the Microsoft presence in the desktop market.

IPSec is stronger but more expensive, requiring agreements with third-party verification authorities. It is also more complex to understand.

The real question, when you are going to chose such a protocol, is how much value do you attach to your tunneling data? The answer to this question should drive your VPN decisions.

An internet site of interest, if you are in the process of picking a VPN is, the website of Bruce Schneier. is a California company specialized, as its name suggests, in Internet security. On their website, you will find detailed cryptographic analysis of both PPTP and IPSec. In both analysis, methods are explained in good detail. Openings and flaws, which may be used to crack these two protocols, are also described.


An AS/400 Perspective:

The AS/400 supports two types of VPN technologies: IPSec and L2TP.

We have already seen IPSec, and I will not go into details again here. L2TP is short for Layer-Two Tunneling Protocol, an extension to the PPP protocol that enables ISPs to operate Virtual Private Networks (VPNs). L2TP merges the best features of two other tunneling protocols: PPTP from Microsoft and L2F from Cisco Systems. L2TP, like PPTP uses CHAP for authentication verification. Increasingly, this is only considered adequate for a small VPN. For larger installations, the choice tends to be IPSec, as it gives more extensive identity scaling capability.

I did not explore L2TP in detail because of its smaller presence in the market. IPSec at time of printing, is the de facto standard, probably because it is a better match for the AS/400's "industrial grade" processing application type than L2TP.



VPN technology is a field of study in itself. In this article, I have just skimmed the surface. This topic encompasses aspects of protocols, encryption, methods of communication, synchronization and security. When I first started researching for this topic, IKE, PKI, PPTP and IPSec were all a jumble of acronyms. Taking the time to put them in proper order was an interesting task and a worthwhile effort. I encourage anyone who has to make this type of decision to take the time to digest this information.

Coming up with new protocols for industrial strength VPN applications take a lot of energy in research dollars, effort and leadership. IPSec alone has been in development for 5 years already.

At this point, Cisco and Microsoft are leading the way with two very different solutions and they lead for two different reasons. Cisco's IPSec on the merit of its techniques and its strength, and Microsoft's PPTP on the merit of its wide spread appeal (it is free and on every NT desktop).

It will take even more energy or perhaps a radical breakthrough to come up with a new and improved method of implementing VPN. More than likely, there will be a number of incremental improvements on the current technology before a completely new generation VPN protocol comes to us.

You can find a soft copy of this article at

Credits & Sources

The IPV6 brief was sourced out on the Internet at the following websites:

I would like to thank Mike Smith, of IBM Canada and Miles Jenkins of ADCT for proofreading this article prior to publication.

Back to Tylogix Home Page