So I have been toying with the idea of 10G networking. Specially since I built a NAS for my needs, so 10Gbe seems like part of my natural progression.
I was looking at this for weeks, but most of the time this looked like difficult unless I was willing to spend quite a bit of cash into this? until I stumbled with this video:
Which kinda suprised me in terms of low implementation complexity. Also, this idea was right around my budget (more on this later). Even my use case and setup was similar, and I immediately knew this was my starting point.
Here is a list of things I consider ?reasonable? for this project:
- Budget: $200?$250 USD. So yes, this is the part where it might be cheap or not. To me, spending this amount is? acceptable. For you it might not. This technology doesn?t really come cheap, specially since in most cases this is not currently oriented towards home environments. Perhaps some day economy of scale will allow the transition to 10G way cheaper than now. Given I don?t live on the US, and I will have to be importing every single component, if I exchange this investment into local currency then yeah, this is hell of expensive. I don?t think I know someone personally willing to spend this on just a home network change. But to me it is worth it because is really educational and I like it. Still, I don?t have unlimited budget, so $250 USD will be all I can spend on this (hopefully less) Not including cargo forwarding to my country.
- Preferably fiber instead of DAC (Direct Attached Copper) because why not. I mean, copper is fine. But to me, it is not interesting. And this an experiment anyway
- The new core switch has to have PoE support. I have an access point that provides Wi-Fi to me and my family. I like that it only requires one Ethernet cable to work, and I?d like to maintain it that way. Besides, I?m thinking on adding at least one more PoE device at some point, so it is handy for me.
- Less than 8 Gbe ports is unnacceptable. Since I?m currently using 7 out of my 8 Gigabit Ethernet ports available, I would like to have some room in case I need to add more devices down the line. Having said that, 48-port switches are just too much. I live in a small apartment, so having one of those bulky, loud switches is a no-go for me, unless? I compromise and do some modding, if needed (I?ll explain later).
- Preferably 4 SFP+ ports. I could make it with only 2 ports, and there are excellent switch options, like Mikrotik CSS326?24G-2S+RM, which can usually be found on eBay for $150 (brand new) or less (used). But to me this is not an optimal alternative. Why would I invest more or less the same to get exactly the minimum amount of SFP+ ports and features? What if have to grow in the future? I need some headroom.
- Low noise. As I mentioned before, I live in a small apartment, so having those kind of hairblower type of switch noise is not acceptable, specially since whatever I will get will sit on a shelve right in the middle of my apartment, and close to a door of another room where one of my children sleeps.
So I needed to first get a picture (literally) of the current state of my network, and design my upgrade from there. Here it goes:
Before
BTW for those curious, my guests run through a separate wireless network, which is not pictured. Close friends and family still run through my standard wireless. I like simplicity, so for the moment, I don?t see the need to run VLANs on such a small network. That way I don?t really have to manage anything like that (for now).
So, in short, this is where I want to get to:
After
And how do I get there? Well, I set myself to do a list of what I need:
- A new switch
- Two 10G network cards
- At least two fiber cables
- At least four SFP+ transceivers
Most of my debate was around choosing a switch. My first option was Mikrotik CRS305?1G-4S+IN. Pros: It is small, it is quiet (passively cooled), within budget, low power, can run with PoE IN and it has four SFP+ ports. Cons: I would have to add yet another device to my network, it would be nice to just have one core switch to do everything (not a big deal). Now more recently I learned (from recent buyers) that when powered by PoE IN port, and all SFP+ ports are used, there may be some issues like performance degradation (maybe even package dropping?), quite possibly because running all that from just PoE might be a challenge, but this is not an issue if you just use its power cord.
My second option was Quanta LB6M. But I discarded this rather quickly. Pros: Plenty of SFP+ ports, price, PoE, manageable (if that is important to you), well documented. Cons: Too big/bulky, noisy, not enough RJ-45 ports and finally, kinda on the power hungry side. If it is within budget is a hit or miss, sometimes you can find it cheap on eBay, or not.
In the end I went with Aruba S2500 (the 24-port version). Pros: within budget (actually I made an offer to eBay seller and I got it even cheaper), healthy number of RJ-45 and SFP+ ports, relatively medium power consumption, PoE, manageable, this single device can do it all for me. Cons: not very well documented, getting to EoL support, not the biggest but definetely not the smallest either and relatively noisy (loud at startup, but then there is still a hum and I need it to be dead silent). This last part made me think about my choosing for a few weeks? until someone uploaded this video:
As you see, it its perfectly possible to mod this switch to make it completely silent. But you must be willing to invest close to $60 USD just on fans! Most fans on this type of devices have a high peak RPM (11,000+) and the Noctua used in that mod are 5,000 RPM (which is part of why they are so silent) To me, this is the proper way to do it, but investing that amount just on fans would mean to possibly break my budget.
So what I did was to order Scythe Mini Kaze Ultra SY124020L which are 3,500 RPM fans used by other people for similar projects. Those are NOT PWM fans, so you will lose the ability to dynamically control the rotational speed, but since those are such low RPM fans, you will want to ignore fan control warnings from switch and then have them running at full speed all the time anyway. For network cards I will be using Mellanox?s ConnectX-2 NIC (some people recommend going with ConnectX-3 though). As transceivers, I chose Finisar FTLX8571D3BCL.
Here is a more detailed list of all the parts, including what I paid for each, totals at the end:
Costs breakdown
Which is right on point.
Ready with all the necessary components:
Now, on to the fan mod. After sanding out the edges of the fan connectors I went ahead and repalced all of them:
NOTE: You wil have to use the included fan adaptors. Otherwise the fan cable will not reach the header on the switch:
I don?t like the included fan screws (the ones from the switch). They?re bad quality.
After this, you will need to connect the first machine. Install SFP+ transceiver into NIC port:
Then secure fiber cable into SFP+ port:
Now do the same on Switch end:
Install NIC on PC and boot it up:
In my case, I did not had to install drivers from Mellanox?s website. Windows detected the device and configured it automatically:
10G NIC automatically configured on Windows 10
To configure the switch, I recommend watching this video:
NOTE: his method for getting into GUI was not successful, so I had to download firmware onto a FAT32 USB stick and then put the file into a folder called ? arubaimage?, then go to front menu and select upload image. Then I went again into front menu, did a factory reset and I could reach the setup GUI no issues.
By this point you will need something to test out your bandwidth. You will have to have one device to act as a server and a second device (also on 10G network) to act as client. I chose iperf3 to do my benchmarking. If you want a cool explanation, go here:
So this is my test before the upgrade (see bold, my NAS as server, using 4 parallel threads):
Connecting to host 10.0.0.70, port 5201[ 4] local 10.0.0.28 port 49962 connected to 10.0.0.70 port 5201[ 6] local 10.0.0.28 port 49963 connected to 10.0.0.70 port 5201[ 8] local 10.0.0.28 port 49964 connected to 10.0.0.70 port 5201[ 10] local 10.0.0.28 port 49965 connected to 10.0.0.70 port 5201[ ID] Interval Transfer Bandwidth[ 4] 0.00-1.00 sec 28.6 MBytes 240 Mbits/sec[ 6] 0.00-1.00 sec 28.6 MBytes 240 Mbits/sec[ 8] 0.00-1.00 sec 28.5 MBytes 239 Mbits/sec[ 10] 0.00-1.00 sec 28.4 MBytes 238 Mbits/sec[SUM] 0.00-1.00 sec 114 MBytes 957 Mbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 1.00-2.00 sec 28.2 MBytes 237 Mbits/sec[ 6] 1.00-2.00 sec 28.1 MBytes 236 Mbits/sec[ 8] 1.00-2.00 sec 28.2 MBytes 237 Mbits/sec[ 10] 1.00-2.00 sec 28.2 MBytes 237 Mbits/sec[SUM] 1.00-2.00 sec 113 MBytes 947 Mbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 2.00-3.00 sec 28.2 MBytes 237 Mbits/sec[ 6] 2.00-3.00 sec 28.4 MBytes 238 Mbits/sec[ 8] 2.00-3.00 sec 28.4 MBytes 238 Mbits/sec[ 10] 2.00-3.00 sec 28.2 MBytes 237 Mbits/sec[SUM] 2.00-3.00 sec 113 MBytes 950 Mbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 3.00-4.00 sec 28.4 MBytes 238 Mbits/sec[ 6] 3.00-4.00 sec 28.2 MBytes 237 Mbits/sec[ 8] 3.00-4.00 sec 28.2 MBytes 237 Mbits/sec[ 10] 3.00-4.00 sec 28.2 MBytes 237 Mbits/sec[SUM] 3.00-4.00 sec 113 MBytes 949 Mbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 4.00-5.00 sec 28.4 MBytes 238 Mbits/sec[ 6] 4.00-5.00 sec 28.2 MBytes 237 Mbits/sec[ 8] 4.00-5.00 sec 28.2 MBytes 237 Mbits/sec[ 10] 4.00-5.00 sec 28.2 MBytes 237 Mbits/sec[SUM] 4.00-5.00 sec 113 MBytes 949 Mbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 5.00-6.00 sec 28.2 MBytes 237 Mbits/sec[ 6] 5.00-6.00 sec 28.4 MBytes 238 Mbits/sec[ 8] 5.00-6.00 sec 28.2 MBytes 237 Mbits/sec[ 10] 5.00-6.00 sec 28.4 MBytes 238 Mbits/sec[SUM] 5.00-6.00 sec 113 MBytes 949 Mbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 6.00-7.00 sec 28.2 MBytes 237 Mbits/sec[ 6] 6.00-7.00 sec 28.2 MBytes 237 Mbits/sec[ 8] 6.00-7.00 sec 28.2 MBytes 237 Mbits/sec[ 10] 6.00-7.00 sec 28.2 MBytes 237 Mbits/sec[SUM] 6.00-7.00 sec 113 MBytes 949 Mbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 7.00-8.00 sec 28.2 MBytes 237 Mbits/sec[ 6] 7.00-8.00 sec 28.2 MBytes 237 Mbits/sec[ 8] 7.00-8.00 sec 28.2 MBytes 237 Mbits/sec[ 10] 7.00-8.00 sec 28.1 MBytes 236 Mbits/sec[SUM] 7.00-8.00 sec 113 MBytes 947 Mbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 8.00-9.00 sec 27.8 MBytes 233 Mbits/sec[ 6] 8.00-9.00 sec 27.6 MBytes 232 Mbits/sec[ 8] 8.00-9.00 sec 27.6 MBytes 232 Mbits/sec[ 10] 8.00-9.00 sec 27.8 MBytes 233 Mbits/sec[SUM] 8.00-9.00 sec 111 MBytes 929 Mbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 9.00-10.00 sec 28.4 MBytes 238 Mbits/sec[ 6] 9.00-10.00 sec 28.4 MBytes 238 Mbits/sec[ 8] 9.00-10.00 sec 28.2 MBytes 237 Mbits/sec[ 10] 9.00-10.00 sec 28.2 MBytes 237 Mbits/sec[SUM] 9.00-10.00 sec 113 MBytes 950 Mbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ ID] Interval Transfer Bandwidth[ 4] 0.00-10.00 sec 283 MBytes 237 Mbits/sec sender[ 4] 0.00-10.00 sec 283 MBytes 237 Mbits/sec receiver[ 6] 0.00-10.00 sec 282 MBytes 237 Mbits/sec sender[ 6] 0.00-10.00 sec 282 MBytes 237 Mbits/sec receiver[ 8] 0.00-10.00 sec 282 MBytes 237 Mbits/sec sender[ 8] 0.00-10.00 sec 282 MBytes 237 Mbits/sec receiver[ 10] 0.00-10.00 sec 282 MBytes 237 Mbits/sec sender[ 10] 0.00-10.00 sec 282 MBytes 237 Mbits/sec receiver[SUM] 0.00-10.00 sec 1.10 GBytes 948 Mbits/sec sender[SUM] 0.00-10.00 sec 1.10 GBytes 948 Mbits/sec receiver
And this is my test after upgrade:
Connecting to host 10.0.0.71, port 5201[ 4] local 10.0.0.45 port 50670 connected to 10.0.0.71 port 5201[ 6] local 10.0.0.45 port 50671 connected to 10.0.0.71 port 5201[ 8] local 10.0.0.45 port 50672 connected to 10.0.0.71 port 5201[ 10] local 10.0.0.45 port 50673 connected to 10.0.0.71 port 5201[ ID] Interval Transfer Bandwidth[ 4] 0.00-1.00 sec 280 MBytes 2.34 Gbits/sec[ 6] 0.00-1.00 sec 278 MBytes 2.33 Gbits/sec[ 8] 0.00-1.00 sec 269 MBytes 2.25 Gbits/sec[ 10] 0.00-1.00 sec 265 MBytes 2.22 Gbits/sec[SUM] 0.00-1.00 sec 1.07 GBytes 9.15 Gbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 1.00-2.00 sec 289 MBytes 2.42 Gbits/sec[ 6] 1.00-2.00 sec 287 MBytes 2.41 Gbits/sec[ 8] 1.00-2.00 sec 277 MBytes 2.33 Gbits/sec[ 10] 1.00-2.00 sec 274 MBytes 2.30 Gbits/sec[SUM] 1.00-2.00 sec 1.10 GBytes 9.45 Gbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 2.00-3.00 sec 289 MBytes 2.43 Gbits/sec[ 6] 2.00-3.00 sec 287 MBytes 2.41 Gbits/sec[ 8] 2.00-3.00 sec 273 MBytes 2.29 Gbits/sec[ 10] 2.00-3.00 sec 270 MBytes 2.27 Gbits/sec[SUM] 2.00-3.00 sec 1.09 GBytes 9.40 Gbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 3.00-4.00 sec 290 MBytes 2.43 Gbits/sec[ 6] 3.00-4.00 sec 289 MBytes 2.42 Gbits/sec[ 8] 3.00-4.00 sec 270 MBytes 2.26 Gbits/sec[ 10] 3.00-4.00 sec 268 MBytes 2.25 Gbits/sec[SUM] 3.00-4.00 sec 1.09 GBytes 9.37 Gbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 4.00-5.00 sec 286 MBytes 2.40 Gbits/sec[ 6] 4.00-5.00 sec 286 MBytes 2.40 Gbits/sec[ 8] 4.00-5.00 sec 265 MBytes 2.22 Gbits/sec[ 10] 4.00-5.00 sec 264 MBytes 2.22 Gbits/sec[SUM] 4.00-5.00 sec 1.08 GBytes 9.23 Gbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 5.00-6.00 sec 290 MBytes 2.44 Gbits/sec[ 6] 5.00-6.00 sec 290 MBytes 2.43 Gbits/sec[ 8] 5.00-6.00 sec 270 MBytes 2.27 Gbits/sec[ 10] 5.00-6.00 sec 269 MBytes 2.26 Gbits/sec[SUM] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 6.00-7.00 sec 284 MBytes 2.38 Gbits/sec[ 6] 6.00-7.00 sec 284 MBytes 2.38 Gbits/sec[ 8] 6.00-7.00 sec 265 MBytes 2.23 Gbits/sec[ 10] 6.00-7.00 sec 264 MBytes 2.21 Gbits/sec[SUM] 6.00-7.00 sec 1.07 GBytes 9.20 Gbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 7.00-8.00 sec 287 MBytes 2.40 Gbits/sec[ 6] 7.00-8.00 sec 286 MBytes 2.40 Gbits/sec[ 8] 7.00-8.00 sec 266 MBytes 2.23 Gbits/sec[ 10] 7.00-8.00 sec 265 MBytes 2.22 Gbits/sec[SUM] 7.00-8.00 sec 1.08 GBytes 9.26 Gbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 8.00-9.00 sec 291 MBytes 2.44 Gbits/sec[ 6] 8.00-9.00 sec 290 MBytes 2.43 Gbits/sec[ 8] 8.00-9.00 sec 272 MBytes 2.28 Gbits/sec[ 10] 8.00-9.00 sec 270 MBytes 2.26 Gbits/sec[SUM] 8.00-9.00 sec 1.10 GBytes 9.42 Gbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ 4] 9.00-10.00 sec 281 MBytes 2.36 Gbits/sec[ 6] 9.00-10.00 sec 279 MBytes 2.34 Gbits/sec[ 8] 9.00-10.00 sec 265 MBytes 2.22 Gbits/sec[ 10] 9.00-10.00 sec 262 MBytes 2.20 Gbits/sec[SUM] 9.00-10.00 sec 1.06 GBytes 9.12 Gbits/sec- – – – – – – – – – – – – – – – – – – – – – – – -[ ID] Interval Transfer Bandwidth[ 4] 0.00-10.00 sec 2.80 GBytes 2.40 Gbits/sec sender[ 4] 0.00-10.00 sec 2.80 GBytes 2.40 Gbits/sec receiver[ 6] 0.00-10.00 sec 2.79 GBytes 2.40 Gbits/sec sender[ 6] 0.00-10.00 sec 2.79 GBytes 2.40 Gbits/sec receiver[ 8] 0.00-10.00 sec 2.63 GBytes 2.26 Gbits/sec sender[ 8] 0.00-10.00 sec 2.63 GBytes 2.26 Gbits/sec receiver[ 10] 0.00-10.00 sec 2.61 GBytes 2.24 Gbits/sec sender[ 10] 0.00-10.00 sec 2.61 GBytes 2.24 Gbits/sec receiver[SUM] 0.00-10.00 sec 10.8 GBytes 9.30 Gbits/sec sender[SUM] 0.00-10.00 sec 10.8 GBytes 9.30 Gbits/sec receiver
No jumbo frames enabled (yet), no manual driver installation, no issues!
){kind=link}
){kind=link}