Have standalone locators and network servers as you have. But as Eric mentions, you need to have network servers running on the locators so that the client can talk to one of those and connect to a server in a load-balanced way, and failover when required. The client only needs to mention the host:port of netserver running on one of the locators, and the driver will internally redirect the data connection to the proper server (the locator connection is only a control connection to fetch server information which itself will be redirected to a new locator if current one fails). Also you cannot use the peer-discovery-port for client connections as Eric has mentioned (use client-port).
Alternatively you can connect to the data servers while running the locator as you have been successful in doing. However in this case, for each connection the server has to in turn ask the locator about load-balancing/failover information for the client, so a two hop for every connection. This is not recommended even otherwise, since if multiple servers go down, the information on clients about available servers may become stale (and so failover may sometimes fail even with servers available). The locators, on the other hand, have no data and very little activity and deployment should ensure that those are unlikely to fail (redundancy among locators themselves will alleviate possibility of locator failures in production).