TCP/IP - warts and all

Him Her

VIP Member
VIP Member
Joined
Dec 23, 2011
Messages
8,145
Reaction score
6,435
Location
North Yorkshire
No apologies for starting at the beginning gurus - chapter one is supposed to encourage people to have a go, not totally freak them out :)



In the beginning

Once upon a time there were many protocols but the advance of the Internet means they have largely fallen by the wayside and IP has become the protocol of choice. Designed to be scalable from the outset, IP is really a suite of protocols with TCP being one. Others include UDP and ICMP.

As the cost of communications technology has fallen, the Internet has spread and more people have multiple technology types in the home. The home network has become more complex than it ever was and a better understanding of how networking works may be useful.

This is a practical look at how stuff works rather than an academic exploration. When you're up to your wotsits in network cables and communications kit that doesn't work you won't be reaching for a well-thumbed copy of “The OSI 7-layer Model Explained”!

IP in perspective

IP networking was conceived decades ago, originally so the Pentagon could reliably bomb places like Russia. However, while it has been continually updated, some of the original remnants of the old technology rear their heads from time to time. It is useful to know about this as it explains some stuff that is otherwise incomprehensible.

When IP was developed, communications equipment was configured by specialists writing in machine code – the basic computer-speak – almost at binary level. Code had to be efficient so it would run quickly and some instructions run faster than others. Built into IP are apparent anomalies that are explained when you look at the binary. Such as what the subnet mask does, why you can't assign an IP address of 192.168.0.0 and why the so-called Classes have unusual number boundaries.

First though, a quick picture.

Network-01.jpg

It's a flowchart for what happens when you try to connect to summat by name over an IP network. The name gets resolved to an IP address using either DNS (Dynamic Name Service) or by doing a lookup on the local HOSTS file. The IP address of the system you are trying to connect to will have been provided either by DHCP or manually (Static). You don't have to do the name resolution bit if you know the IP address already.

ARP (Address Resolution Protocol) is used to resolve the MAC address and, finally, something gets sent.

If you're sending to the Internet or anything on a different network (even if it's in your house) there's another stage – routing.

When a device attempts to send, it needs to know if the target is on the same network as itself or not. It does this by comparing its own IP address with the target's IP address and uses the subnet mask to get rid of uninteresting bits. That's what the subnet mask is for.

Same network example:

Sender IP 192.168.0.10
Target IP 192.168.0.20
Subnet mask 255.255.255.0

The subnet mask says ignore the last octet (8 bits, the IP address comprises 4 octets separated by dots). The 255 means check this, the 0 means forget it. A quick check shows that 192.168.0 is the network part and the same for both source and target so the packet gets sent direct.

Different network example:

Sender IP 192.168.0.10
Target IP 192.168.10.20
Subnet mask 255.255.255.0

Same subnet mask so we're only interested in the first three octets and this time 192.168.0 is clearly not equal to 192.168.10 – the target is on a different network. In this case send the packet to the default router.

We therefore need three key numbers to make our network do something, IP address, subnet mask and default router address. Actually, you only need the default router address if you want to connect to another network, like the Internet, but let's assume we will always want to do this. All that lot is IPv4, forget IPv6 for now as unless you're working in the IT industry you'll likely not need to worry about it for years.

[Hope it's useful - let me know before I post more :) ]
 
I'll take the likes as a go for chapter two...

Make mine a CIDR

Originally, there were classes i.e. groups of IP address ranges that were set aside for particular uses. They've largely been superseded but people still refer to them.

Class A

The range is 0. to 127. in the first octet
Aimed at the largest organisations
One octet used for the network number, three octets for the device number
Some are reserved such as 0. (the World), 10. (private) and 127. (loopback)
Subnet mask is 255.0.0.0

So a typical address would be 80.0.0.1 with 80. being the network number and 0.0.1 being the device (computer etc.) A private network is not routable over the Internet (because the ISP drops the packets). Loopback is used for diagnostics and is automatically assigned – 127.0.0.1 is the loopback address of your first network card.

A small number of networks can be defined but each can have 256*256*256 (16,777,216) devices (in theory).

Class B

The range is 128. to 191. in the first octet
Aimed at mid-sized organisations
Two octets are used for the network number, two octets for the device number
Some are reserved such as 172.16.-172.31. (private) and 169.254 (auto-config)
Subnet mask is 255.255.0.0

A typical address is 163.1.0.8 with 163.1. being the network number and 0.8 being the device. More networks but fewer devices (65536).

Class C

The range is 192. to 223. in the first octet
Aimed at smaller organisations
Three octets are used for the network number, one octet for the device number
Some are reserved such as 192.0.0.-192.255.255. (private)
Subnet mask is 255.255.255.0

A typical address is 193.0.10.4 with 193.0.10. being the network and .4 being the device. Only 254 devices can be attached but there are some 16 million networks available.

More classes exist but they're not so interesting.

We've mostly moved to CIDR now – Classless Inter-Domain Routing – whereby you can use any numbering and apply a subnet mask to suit. So the old class C network would be 192.168.0.0/24 with the /24 simply meaning the subnet mask is 255.255.255.0 (255=8 bits, three lots of 255=24bits) although other combinations are allowable. For example, you could use 192.168.0.0/16 where the subnet mask would be 255.255.0.0 thus giving it the appearance of a class B network in some respects, but more of that later.

Some simple, or older, equipment may still reject CIDR notation.

Wrap your data in packets

You can't just launch raw data onto the network – it has to be in a packet. This is a bit like the postal system, imagine the packet as being like an envelope that has to be signed for. The packet has three main components - head, body and tail. The head contains information such as where it came from (IP address and port) and where it's going (again, IP address and port). The tail is the signed for bit with checking to ensure the data are intact. The body is your data.

Although your computer usually has just the one IP address this does not uniquely identify the sender, the port is used for that. Run netstat -a to take a look at the open ports on your computer. An open port means something is listening. Most people are aware of ports from a server point of view i.e. web servers are on port 80 normally but if your browser didn't use ports as well as the IP address then the returning information (dodgy download or whatever) couldn't find it's way back. Open a new tab in the browser and it gets a new port.

There are 65535 TCP ports available so you shouldn't run out!

There are 65535 UDP ports too so what's the difference?

TCP (Transmission Control Protocol) is connection-oriented which simply means it checks delivery occurred correctly and does something if it went wrong. Web browsers use TCP. UDP (User Datagram Protocol) is connectionless so doesn't do any checking, it's the transport mechanism for TCP and used directly by some services such as TFTP (Trivial File Transfer Protocol, used to flash firmware over the network). ICMP is used by diagnostic tools such as ping.

An understanding of ports is useful when trying to figure out how things actually work.

Port example:

PC with IP address 192.168.0.2 connects to DW using a browser
DW is listening on port 80 so the PC sends to that port
PC opens a local port (16454, say) to listen for a response and tells DW
DW server checks the sending port and replies
PC is listening on port 16454 so it gets the response

The Internet is NOT in your house!

Well, obviously. But that means you have to route off your network and onto the Internet. Routing is a two-way process, your PC has to know how to get to another network and it uses the default router address. Whenever it gets a request for an IP address not on the local network it ARPs for the default router and sends the packet there instead.

Assuming routing is working properly, the other end knows how to get back to you. If you're experimenting with routing with your own networks then you need to have two or more distinct subnets connected by a router. The two (or more) devices that need to connect should have a default router address set and it should be from their perspective i.e. the default router should be on the same subnet as the sender.

Example:

PC1 – IP address 192.168.0.2, subnet mask 255.255.255.0, default router 192.168.0.1
PC2 – IP address 192.168.1.2, subnet mask 255.255.255.0, default router 192.168.1.1
Router interface 1 is 192.168.0.1, subnet mask 255.255.255.0 and is connected to PC1
Router interface 2 is 192.168.1.1, subnet mask 255.255.255.0 and is connected to PC2

The subnets are different so routing can happen. If the subnets are the same routing can't happen.
 
I'm tied up for a day or two so if you can't sleep...


Any port in a storm

One of the problems with IPv4 is the limited number of addresses available – a shade over 4 billion – and for this reason IPv6 was developed. In the meantime NAT/PAT was developed as a workaround. The problem is, it's expensive to move from IPv4 to IPv6 so it will happen more gradually and NAT/PAT will plug the gap for a while perhaps even spilling over into IPv6 as there are some useful security features in it.

Before NAT/PAT you would need a public IP address for every device you wanted to connect to the Internet while now you 'hide' behind a relatively small number of public addresses, or even one. In the case of home users usually one address suffices. It's all done with ports!

The box you connect to the Internet with runs NAT/PAT by default, it works like this...

The ISP assigns the box all the relevant IP address information including public IP, subnet mask, default router and DNS (for name resolution)
Your PC gets a different IP address from the router, usually via DHCP and typically a 192.168.x.x network
Your PC tries to connect to a web server and figures out it's not on your network so sends the packet to the default router (the ISP box)
The default router has a NAT/PAT table of around 5000 entries typically and records where the request came from including IP address and port
It then sends the request to the web server on your behalf
The web server sees the request as originating from the router and sends the reply back
Your router looks up the original sender in the NAT/PAT table and forwards the reply back to you

Voila! You hide multiple devices behind fewer external IP addresses thus fooling the web server into thinking there's just one device.

The outcome is less strain on diminishing numbers of available IPv4 addresses and less need to worry about IPv6 for a while.

Mostly, home systems utilise PAT but if you add some NAT elements in you need to be aware of the risks.

When you initiate a connection to the Internet using PAT an entry is made into the state table to allow 'stateful inspection' of any replies received. Or, simply put, your router checks the sender is the system you connected to and not just some passing exploit.

Random external callers get blocked because there's no entry in the state table.

However, if you open a port you're telling the router to allow any traffic in on that port and redirect it to whatever IP address and port you specify. Stateful inspection is inactive for these connections and there may be security implications. Of course, if you want to allow the traffic in there's no choice.

When opening a port you may have to specify both TCP and UDP depending on the service type. A quick 'Google' will usually indicate which port(s) and protocols are used.

Binary and other painful things

Earlier I mentioned that sometimes the early technology pokes it's ugly head through our pleasant little GUI world.

In the early days of setting the classes routers were fairly brainless and slow so it was important to grab any efficiencies you could. Class A numbers go from 0. to 127. on the network side while Class B numbers go from 128. to 191. Why not just say 0 to 99, 100 to 199 etc. - it's easier!

The reason is that while we humans see the nice dotted-decimal notation of, say, 10.x.x.x the computer 'sees' 00001010.x.x.x and that means there's a zero in the most significant bit of the network octet – hold that thought!

If we look at a Class B network of, say, 163.x.x.x the computer 'sees' 10111111.x.x.x and now there's a 1 in the most significant bit.

Computer machine language has some fast instructions and some slow ones – speed is determined by how many clock 'ticks' elapse before you get the answer. The fewer the 'ticks' the quicker the result. Rotate Left (ROL) is very fast and simply moves the most significant bit into a flag that can be rapidly tested. Like this pseudo-code example...

Load first octet
Rotate Left
If flag=0 then class=A, do something
If flag=1 then class=B, do something different

Very fast!

For other classes additional Rotates quickly allow you to determine the class of the address range. This doesn't work very well with decimal range breaks as they're harder to test and much slower.

A curious rule of thumb

Because you don't want to be reading some tedious text while pursuing your networking goal just commit this to memory:

In many cases the first and last are unusable!

On this network, 192.168.0.0/24 (subnet mask 255.255.255.0) there are 256 addresses (0-255) but only 254 can be assigned. The reason is that the first (192.168.0.0) is the cable address and must not be assigned to a device while 192.168.0.255 is the broadcast address. The first address and last address of any subnet cannot be assigned to a device – they're unusable.

This crops up time and again.

Broadcasting yourself

All traffic is equal but some is more equal than others!

Some traffic is routable i.e. it can get off the network to another one, like the Internet. Some traffic is not routable but still plays an important part. For example, DHCP is a broadcast technology and not naturally routable.

Broadcasts go to all devices as the target address is set to the broadcast address. In our example above the broadcast address is 192.168.0.255 and all devices, on seeing this address, check the contents. Non-broadcasts are directed to a specific address so while all devices look at what gets sent, as soon as they see it's not addressed to them they drop it.

If you have a home network with your ISP router set to give out IP addresses using DHCP and you add a router internally with a second subnet you need to tweak stuff.

You can either use static addresses for the new subnet or configure the internal router to pass DHCP traffic – sometimes known as DHCP helper or relay.

A nice layer cake

I suppose we'll have to include the OSI 7-Layer Model, at least a bit! Mainly interesting because it specifies what should happen at different levels, or layers, from hardware through to application.

Turn it on it's head for now and just look at layers 1, 2 and 3. It's where you'll mostly come unstuck anyway – or in my terms, get this right and the packet comes out the wall, after that it's not my problem!

Layer 1

Physical. We're talking plugs and bits of wire, socket types, current flows and voltages. Some devices may live here such as repeaters (hubs), not too bright stuff, routing flies over its head.

Layer 2

Data Link. Knows about MAC addresses and logical links. Lower protocols like Ethernet and PPP live down here. Devices such as simple switches live here too.

Layer 3

Network. Knows about routing and IP addressing. Higher protocols such as TCP and UDP live here along with cleverer switches than can route.

The layers interact by wrapping or unwrapping the information – a bit like a series of envelopes. So PC1, sending to PC2 constructs an IP packet at Layer 3 and passes it to Layer 2 which encapsulates the packet in a frame (envelope) – typically Ethernet. This is passed to Layer 1 which does the signalling on the wire.

PC2 receives the signal and converts it back to a frame which it passes to Layer 2. Layer 2 unwraps the frame to reveal the payload – the original IP packet with your data inside – which it passes to Layer 3 and so on.

And that's the point really, if you change something in a layer – like change to a different network card – then you only need to change stuff on your system at the affected layer(s). You don't re-install Windows to accommodate the new hardware – you just install a driver to link into the layers above.

[All is cool, all interfaces within normal parameters...]

:)
 
Last edited:
Do I dare try to explain subnetting - when everyone else, EVER, has made such a total c*ck-up of it???
 
I personally think that you've done a great job in breaking down the fundamentals of TCP/IP.

There is a always a distinct danger of information overload as far as this subject goes, but if you've whetted anyone else's appetite for this, and have given them at least a basic understanding of it's going on, then this thread has been more than worthwhile.

It's entirely up to you, but I would maybe touch on the basic ports (21, 25, 80, 110, 443 etc) and describe how they fit in to the grand scheme of things, then take it from there.

Great work.
 
Good point - an example of actually using ports may be useful. A list of common port usage and the services that sit on them. For example, lots of peeps get their mail over POP and if you get a crappo message stuck it's a pain but you can destroy the block yourself using telnet to port 110.

Thought of looking at common exploits too such as DoS (Denial of Service) etc.and how they can be killed ...
 
Best post on DW for a looong time.

1 question, do unique IP addresses get added to a computers HOST file? i.e once we know the IP address of an WWW site does our computer then store this in the HOST file so we no longer need to route through the DNS?
 
Best post on DW for a looong time.

1 question, do unique IP addresses get added to a computers HOST file? i.e once we know the IP address of an WWW site does our computer then store this in the HOST file so we no longer need to route through the DNS?

no but you can add it by editing the hosts file c:\windows\system32\drivers\etc or Unix /etc/hosts

When you go to say www.someplace.com - it does a ns lookup on the hostname to get an IP address. This is then stored in dns cache - so when you go to same website, no need to do a ns lookup as it has the IP stored. To see your DNS cache in Windows, in command prompt type

ipconfig /displaydns

As far as I am aware in Unix you cant view your DNS cache, but if you type

/usr/sbin/nscd -g

you can see statistics (nscd is the Name Service Caching Daemon)

In Windows you can clear dns cache by typing in command prompt

ipconfig /flushdns

In Unix you need to restart the nscd daemon service to flush the cache.

service nscd restart
 
I don't believe viewing the DNS cache on Linux, in its standard form, is possible although third-party (DNS) utilities may provide that function. Linux doesn't automatically cache anyway and often leaves that to the application so nscd may not even be running/installed.

You can check by:

ps aux | grep nscd (command may vary slightly between distros)

If nscd IS running you'll see a reference with a process ID (pid) and where the binary loaded from.
 
Last edited:
Best post on DW for a looong time.

1 question, do unique IP addresses get added to a computers HOST file? i.e once we know the IP address of an WWW site does our computer then store this in the HOST file so we no longer need to route through the DNS?

One of the reasons it doesn't do this is because the HOSTS file is a simple text file and not indexed (like a database) therefore as it gets longer lookups really start to slow down.

It's a throwback to before DNS was deployed and used as a standby now where DNS may not be appropriate but it takes some management as you have to update ALL the relevant HOSTS files on a given network. DNS lookups should be fast as it is distributed so DNS zones are spread throughout the Internet. The ISP often provides some caching too speeding lookups and if a given DNS server fails to resolve locally it will refer to a 'root hints' server which basically says 'for this domain - look over there'.
 
Why not play with my ports?

As others have remarked on here, getting your hands on is the best method! So, why not play with my ports...

Of the 65535 ports mentioned earlier, some of them fall into the 'well-known ports' class. That is, common services are associated with them such as web servers that live on port 80 by default and e-mail is typically delivered to port 25 (SMTP). E-mail can be collected on port 110 (POP) or 143 (IMAP). Some ISPs may change these ports for security reasons.

The ports in common usage are maintained by IANA (Internet Assigned Numbers Authority) and you can get the full details here. Briefly, we are concerned only with the ports numbered 0 through 1023 as that's where the most common services live.

Now, knowing what port a service is on not only allows you to connect to it, it can also allow some diagnostics and testing using Telnet. There's plenty of stuff about Telnet so I'll assume everyone has some form of client (Windows/7 users will have to install one but there are plenty of free ones about).

Telnet normally connects to port 23 where it expects to find a terminal service to which it can log-on and issue commands. However, that's just the default, Telnet can connect to ANY port. You just need to know what to type to get some kind of meaningful response.

One thing you can do is test e-mail services. If you connect to port 25 you can test e-mail sending, connecting to port 110 allows testing of POP e-mail receiving.

It's easier to understand this if you have a server to connect to and the ISPs don't usually like it if you mess with theirs! No matter, you can connect to mine.

The following examples are 'live' commands so you can type them into your Telnet client and see real results.

When you send an e-mail, one way or another, an SMTP service gets invoked. The commands that SMTP will respond to are detailed in an RFC (Request for Comments), the documents that drive Internet development. A quick Google leads you to an unbelievably tedious reading experience. However, we only need to know the commands to send an e-mail and these are:

HELO <SP> <domain> <CRLF>
MAIL <SP> FROM:<reverse-path> <CRLF>
RCPT <SP> TO:<forward-path> <CRLF>
DATA <CRLF>
QUIT <CRLF>

Where <SP> means press the <SPACE> bar and <CRLF> means press the <ENTER> key.

Telnet to SMTP example:

Run up Telnet and connect to aaland4.demon.co.uk on port 25
This prompt should appear 220 aaland4.demon.co.uk ESMTP Postfix (Ubuntu)
Telling you I'm running an ESMTP service using Postfix on Ubuntu
Type helo engineer <ENTER> to get 250 aaland4.demon.co.uk
The session has been started
Type mail from:<[email protected]> <ENTER> to get 250 2.1.0 Ok
You told the system this e-mail is from [email protected]
Type rcpt to:<[email protected]> <ENTER> to get 250 2.1.5 Ok
You told the system to send the e-mail to [email protected]
Type data <ENTER> to get this prompt 354 End data with <CR><LF>.<CR><LF>
Type a short message followed by <ENTER>, then a dot (.) then another <ENTER>
You'll get a message like this 250 2.0.0 Ok: queued as 27CAFC4A0D0
The number on the end changes each time
Finally type quit <ENTER>

You sent an e-mail using Telnet! On the other hand how do you know?

I've set up browser-based e-mail on the server too so you can connect by following this URL:

http://www.computermedics.co.uk/squirrelmail

Login ID is engineer
Password is letmein

Now you should be able to see your e-mail!

The same trick works with e-mail collection and it's sometimes handy to know because if you get a corrupt message in your inbox the e-mail client may hang and refuse to collect any e-mail at all. Using this information you can connect to your mail service and delete the first message. This time the relevant commands for POP e-mail connections are:

USER <SP> userid <CRLF>
PASS <SP> password <CRLF>
STAT <CRLF>
LIST <CRLF>
RETR <SP> msg# <CRLF>
DELE <SP> msg# <CRLF>
RSET <CRLF>
QUIT <CRLF>

Telnet to POP example:

Run up Telnet and connect to mail.computermedics.co.uk on port 110
You should get this prompt +OK Dovecot ready.
Type user engineer <ENTER> to get +OK
Type pass letmein <ENTER> to get +OK and you're logged on
Type list <ENTER> and the system shows a list of e-mails by number and size
Type stat <ENTER> and it tells you how many e-mails and how much space is taken up
Type retr 1 <ENTER> to view the first e-mail
If you type dele 1 <ENTER> you mark e-mail 1 for deletion on quit
If you marked something wrong type rset <ENTER> to clear ALL deletes
Typing quit <ENTER> logs you out and disconnects

Notes:

1. Each user can have only one concurrent log-on to POP, so if you can't get in someone else is probably connected
2. My SMTP service rejects many countries so if you're in e.g. Russia the first example won't work!
 
Telnet Client is not installed by default on Windows 7 or Windows Vista.

To install Telnet Client on Windows 7 or Windows Vista

Click Start, and then click Control Panel.
On the Control Panel Home page, click Programs.
In the Programs and Features section, click Turn Windows features on or off.
If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue.
In the Windows Features list, select Telnet Client, and then click OK.
 
Telnet Client is not installed by default on Windows 7 or Windows Vista.

To install Telnet Client on Windows 7 or Windows Vista

Click Start, and then click Control Panel.
On the Control Panel Home page, click Programs.
In the Programs and Features section, click Turn Windows features on or off.
If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue.
In the Windows Features list, select Telnet Client, and then click OK.

Aint that weird when later MS stuff requires you to use the Powershell lol
 
Aaaaagh! Hoist by my own security policy lol

It's done a check, found the domain doesn't exist and duffed you out!

I think it's cleared now but you can use any valid domain after the @ if it doesn't work :)

(Only on the mail from bit, rcpt to has to be as written...

I can't test it because I've set internal connections to pass by default...
 
Back
Top