So I've got the 2nd interconnect bay working with info i've got working through searching the web! Here is what happened. When I took over the IT position at my company the C3000 and EMC SAN were already in place and configured and I had no documentation or explanation of the install, to be honest I had no clue how the setup worked anyway. In moving to the new SAN (and asking a lot of questions) I found out the following:
1. The X-Connect ports on both interconnect bays (17 and 18) where in separate VLANS (17 was in the default, and 18 was in the iSCSI VLAN). They also were not in a Trunk group. So i moved them to the default VLAN, added them to the 1st Trunk group. I then turned on VLAN tagging and moved them both to the default VLAN and the iSCSI VLAN so that they could transfer traffic between the two bays on those two VLANS. This now allows my ESX blades to see any iSCSI connections on the 2nd interconnect bay.
2. Ports are hard wired to the individualize bays and cannot be changed.
3. For true redundancy I would have to use one of the Mezzanie slots NICs on the iSCSI network. This would allow the 1st interconnect bay to fail and iSCSI traffic to still flow through the 2nd interconnect bay.
4. Only the two on board ports have a separate iSCSI offload engine. The two NICs in the mezzanine do not include the iSCSI offload engine and will more than likely need to be set as either iSCSI or regular net traffic (still researching this). This is different than the onboard NICS which appear to be able to handle both at the same time. In ESX 3.5 the onboard NICs show up as 2 MAC address/VMNICS, but they show up as 4 MAC address in the Onboard Admin for the C3000. The 2nd set is labeled iSCSI so I assume that once iSCSI traffic hits the NICs the offload engine takes care of it while regular traffic flows through the NIC.