Design Question about Data Structure and Transport Method
Hi everyone,
I'm building an application that will need to keep track of a large list of files (with some meta-data with it) and will have to transfer the list of files to other peers on a local network. I will also need to be able to transfer significant amounts of binary data between peers on the local network. Does it pay for me to write my own protocol over sockets? Or should I use xml-rpc for sending the list/string data and something like ftp for the binary data? FTP might be difficult because the program will need to act as an ftp server and client so that it can send/receive binary data from other peers. Perhaps an HTTP server?
Any ideas?! Thanks!
[683 byte] By [
mjacobsa] at [2007-10-3 3:01:58]
Do you not have any requirements? What you have asked is very vague. Are you writing apps for client and server? Are you pushing or pulling the data? Any restrictions? Any reason why you would want to write your own protocol instead of just using sockets? Are there any technologies you would prefer to use? etc.
Ted.
Hi Ted, thanks for answering. Sorry for being vague. The program is going to be operating on every computer on a local network and is going to be used to share specific files between the machines (which will all already know about each other.) One peer will request a list of files from another, and can then request one of the files from that peer. When I said my own protocol, I meant using sockets with static commands so that client/server sockets know what to do. However, I was wondering if sending binary data files and a large amount of text (for the file list) over a socket connection isn't as efficient as some already existing API or something. I will need to keep track of the files using a file list in java, and I'm not sure what kind of data structure (e.g. linked list, hash table, vector, etc.) is the best to use yet either.
Thanks again!
There's certainly no need to roll your own protocol; there's plenty of established protocols for transferring files. In fact, I would guess that you could pretty much script this using the FTP protocol. It sounds like you're trying to write a fully-featured application when a script would do.
Brian
But to use FTP, wouldn't I need to set up an FTP server in the program as well? What API could I use to instantiate an FTP server?Thanks!
> But to use FTP, wouldn't I need to set up an FTP
> server in the program as well? What API could I use
> to instantiate an FTP server?
No, you could install an actual FTP server on each machine. There's plenty of free ones. The script would simply communicate between all the FTP servers. All you're really doing is asking for a file list (which an FTP server can provide) and the actual file (which it can also provide). Why do you think you need your own FTP server?
Brian
I was looking for a more lightweight solution, I was hoping not to have to run a separate service. I was thinking there might be a very simple ftp server library/api that I would be able to use. I need the application to be a little more robust than a script would be I think-- that's why I'm using java.
And your Java FTP server implementation would be more "lightweight" than what's out there? I suppose you could try using TFTP, though I don't think it supports listing files. You need most of the FTP implementation. True, you don't need to create directories (or maybe you do?), but you certainly need to know when a command failed or the FTP connection has been closed.
As to scripts not being robust, well I sure hope they are because my automated backup processes are scripted. Robustness is simply a matter of ensure that all possible execution paths are handled. Your program logic does not seem to be that complicated to me, therefore a script could easily be robust enough. And I use the term "script" loosely. I mean that the logic is simple enough to perform it sequentially in a single main() method.
while (true) {
listOfFiles = GetListOfFiles();
fileIAmInterestedIn = processListOfFiles(listOfFiles);
if (fileIAmInterestedIn.isThere()) {
if (!transferFile(fileIAmInterestedIn)) {
System.err.println("WARN/ERROR: transfer of " + fileIAmInterestedIn + " failed.");
}
}
Thread.sleep(10000);
}
Another reason I don't want to use an external ftp server is because some systems will be windows and some will be mac (and possibly linux), so I'd like to have a uniform solution instead of installing ftp servers on different platforms, it can get more complicated. Is there another way to do it? Are there any APIs or libraries for easily transferring files between 'peers'?
Thanks again
I don't understand why you see the FTP server solution as more complicated. You write less code, and all of the tricky networking code is already neatly bundled up into a series of FTP commands. You can open connections using URLConnection in Java, and transfer files by using the getInputStream() method of URLConnection. You will not get around requiring a server of some kind. What can you possibly do when you write this server that will make it LESS complicated than using an existing solution?
No matter what you do, there will have to be a server running on the machine. Your Java code would have to do everything the FTP server does (accept connections, clean up old connections, respond to unexpected network conditions, open files, send files over socket, send status codes to client). Yes, you could write your own server with your own protocol, but now you've re-invented the wheel. You could use RMI instead of FTP, but then you need rmid running on every machine.
I really doubt that you will write a better or more robust server than what is freely available. I doubt that you will find a better solution than installing an FTP server on all required machines. But, if all you want to do is get paid to write unneccessary code, then I suggest you learn about RMI and do an RMI implementation. I leave your business case for this solution to you. I'm not very good at weasel words.
Brian
Thank you for your response. Perhaps I should have mentioned earlier, but this is not a project I am working on for any commercial application. I am a college student and this is an application to be used by myself and my housemates this coming year. The reason I did not want to use an external ftp server was for the two reasons:
1. I want to learn how to write a network application
2. I want everything to be in one package for myself and for my friends.
I understand it might be a little more work, but to me it seems more elegant and I've never worked on a project at this level before, so it is exciting to me.
So thank you again for your response, but I am not looking to 'weasel' anyone.
In that case. All you'll need is the socket tutorial (http://java.sun.com/docs/books/tutorial/networking/sockets/index.html). I guess you'll want something like this...public class MyFunnyApp
{
public listen()
{
loop
{
listen for connection
while (more files to read)
{
read java.io.File
}
}
}
public send()
{
open connection
send java.io.File
}
}
And stick buffering in where you want it.
Ted.
