I am no network super-genius but I do enough with networking to be able to get by. Two common mistakes I find many times are flat networks and firewalls as the default gateway. A flat network is when generally switches are connected to one another without any configuration. There is one broadcast domain which means every packet that the switch does not have an entry in the MAC address table, is sent out all the ports but the originating port. This repeats across all of the switches until the layer 2 destination is found. Now, this means your expensive Cisco switches are barely better than hubs. You don’t have collisions like you would on a hub and once the switch learns where the MAC address lives it keeps that information for a certain amount of time. Then again in this network setup the logs are most likely not monitored so if there where collisions and other errors it goes unnoticed.
That is not the title of this post though. Although related to a flat network using the firewall is a different issue. Using the firewall as the router works just fine when you have a flat network. You may never notice the problem in a small network, but as your network grew you noticed how problems can come up when there is just one big network. So someone smart said use vlans to segment the network, create smaller broadcast domains. Then when you try to fix or change the flat network with subnets and vlans can you find out the new vlans can not reach the rest of the original network.
The current flat network with switches and the firewall used as the default gateway or router.
The problem comes when you add subnets that are different than the interface ip of the firewall. Firewalls in general have issue with redirecting traffic bound for other networks back out of the same interface. So in the picture above traffic from vlan 1 that is using the firewall as the default gateway trying to reach the subnet on vlan 10. Since the host on vlan 1 does not know where that network lives it sends the traffic to the default gateway. Even if you added a static route to the firewall the traffic will often fail. That is because firewalls are not meant to route but rather send traffic between trusted and untrusted networks and vice-versa. So the question becomes how do you actually fix your flat network that has the firewall as the router. There is of course more complicated solutions to provide high availability using VRRP or HSRP.
First get a real layer 3 device. That is a router or a switch capable of routing between multiple vlans. The good news is many of your newer switches are capable of layer 3, it is included in many Dell and HP switches, it may still be an add-on with Cisco. I haven’t used a new switch in the last year that did not have layer 3.
Next important step is use the layer 3 device (switch or router) to route everything. Set a default route in the layer 3 device to send only outbound traffic to the firewall and bam everything works. Why is this so hard. Many times there is hundreds of servers and desktops already configured to use the firewall as their router. We will do a lot of work to avoid having to do a bunch of manual work.
Now you are using a router to route and the firewall to block bad things and maybe even do NAT. (note: If you are doing NAT be sure to add your new VLANs to your NAT rules so the new networks can reach the outside of your firewall.)