I'm trying to figure out pNFS, if im not mistaken (correct me if I am please), it has 2 main components: Data servers and Metadata servers, the data servers are conected to the hosts through SAN and the Metadata servers through LAN. The question is, why use two different types of access to the data? I think it adds complexity, it's more expensive and the only reason I can see to justify that is maybe to add performance, but at a very high cost. What would be an example of a good scenario to implement pNFS?
pNFS is designed to allow for a wider set of architectures than you may have initially heard. While it is true that it can be set up with metadata on LAN and data on Blocks(SAN) served by FCP, iSCSI or FCOE, this is just one of 3 architectures available.
Here are the other two.
1. Metadata on LAN and Data(files) on LAN served over NFS4.1 protocol.
2. Metadata on LAN and Data served via objects over OSD2 for example.
Here is a diagram of the 3 architectures. . .
There are different performance and architecture benefits from each architecture, but one overall design goal is performance scalability. As you noted using a blocks and LAN solution adds on-going complexity design and maintenance. Check out the file solution of pNFS implemented with NFSv4.1 for a robust LAN solution.
Here are a couple of good links to find out more if you are curious.