The Anti’s Cloud

Personally, I’ve always been interested in distributed computing. It’s one subject I’ve done a lot of reading on. After seeing a bunch of posts on various Google blogs, such as this one, I’ve decided I want to work on making my own little cloud. I know I don’t have 500,000+ computers to work with like Google, but I do have 5 (or so…I think I can scrap up a few more out of my extra parts).

I’m starting to plan out everything but this would consist of a “manager” computer, and then slave computers to do the actual processing. I’m going to be building a web server specifically for this purpose and using some undecided DBMS (I’m thinking about SQLite).

At least to start with, each request will only be handled by one computer, although I do plan to eventually spread the processing of one request over multiple computers. I’d like to start with smaller goals. The database however will store information in sort of a “striping” manner, where each database computer will have the same structure, data will be separated over multiple computers, but not mirrored. There will be an application inside the manager to layer on top of SQLite (or whatever I decide on) to manage the returned results from all the combined computers.

Again, I’m going to start small and start with a basic web server that serves basic HTML, and then move up and support a dynamic language (Most likely Python or PHP, not sure which yet though, or hell, maybe both). After I get all this working I will make it distributed between computers and reporting to its “manager”.

The only problem with this plan is that it seems like there would be a bottle neck on the “manager” computer. I guess eventually I’ll have to support several of those, and use a load balancer to switch traffic between them.

I haven’t decided what language I’m going to use yet, but I’m leaning towards Python. One might argue that something of this magnitude should be done in a language like C or C++, but the problem is I think those languages would make something like this too complicated, and would require far more code to do less. Personally I think python would be great for the job, however I’m still deciding because I might also want to use C#.

Will definitely keep updating as I work through the project. Feel free to make suggestions on how I should go about things.

An idea for anonymity

I was thinking the other day. How would I do something on the internet and make my activity completely untraceable. The simplest form of the answer was to scatter where the packets come from.

So the idea of the packet scatter network was born. Anyone could sign up and host a server and there would be a large list of servers that the program would use. what the client would do is grab all outgoing packets and scatter them amongst the servers in the list, attaching a forwarding address. when each packet gets to it a server it is then sent on to its final destination, but anyone trying to trace the packets would find anywhere from 2 to thousands of sources.

I have absolutely no idea if this is even vaugley possible. But its and idea. It seems like it would only work for UDP because those are not ordered and such like TCP. But I’m not sure. I’ll ponder on it more I guess. Just an idea.

C# And .dll’s

I’ve been working on Fizzure A LOT recently. I made a FizzSrvLight that is not a distributed system like the regular one, which therefore allowed me to write one effectively in about 3 hours. On the way I decided to make a few of my own methods and then realized, hey these can be used in other projects too!

So I made a class library (.dll – Dynamically Linked Library ) with a few methods that have to do with TCP Data transmition. The most important of which is the Send method that I made. Now this is really only useful for the client. Anyway, heres the snippet:


public static void Send(TcpClient Client, String Command)
{
Console.WriteLine("Opening Server Stream");
NetworkStream n = Client.GetStream();
String send = Command;
String receive = null;
byte[] msg = System.Text.Encoding.ASCII.GetBytes(send);
n.Write(msg, 0, msg.Length);
Console.WriteLine("SENT: {0}", send);
}

this method is meant for console programs, but if you are using a GUI all you really need to do is delete the Console.WriteLines()’s in there and replace it with wherever you want the output.

Hope this is helpful to everyone!

C# TcpListener Example

Ok, this is just a quick snippet of code I wrote to get a working server up. Obviously theres more commands I could put in there in plenty of different ways, but I really just wanted to keep things simple for now. This took me about 2 hours.

This snippet is the main body of code that controls everything. If you go through it and read you’ll see that I made a struct to hold the information on files named File, in the namespace Structure. So you would access it by saying in this [MainNamespace].Structure.File; or you can just use Structure.File. I’ll paste the code for the struct at the end.

I didn’t leave too many comments because I used a lot of Writelines to tell me what it was doing, and for debugging purposes. Those kind of tell you what things do what.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Text;

namespace FizzSrvLight
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine(“FizzSrvLight :: Non-Distributed Fizzure Serving Capabilities”);
System.Threading.Thread.Sleep(1000);
Console.Write(“Loading…”);
Console.WriteLine(“!”);

Console.WriteLine(“Initiating Server Variables…”);
System.Net.IPAddress localaddr = System.Net.IPAddress.Parse(“127.0.0.1″);

Console.WriteLine(“Constructing Server Objects…”);
System.Net.Sockets.TcpListener MainServer = new System.Net.Sockets.TcpListener(localaddr, 9000);

Console.WriteLine(“Starting Server…”);
MainServer.Start();

Byte[] bytes = new Byte[1024];
String data = null;
String send = null;

while (true)
{
Console.WriteLine(“Waiting for connection…”);

// Accept Requests
System.Net.Sockets.TcpClient client = MainServer.AcceptTcpClient();
Console.WriteLine(“Client Connected!”);

// Clear Buffers
data = null;
send = null;

// Get Stream Object for reading and writing
System.Net.Sockets.NetworkStream stream = client.GetStream();

int i;

// Initialize File Holder
System.Collections.ArrayList CurrentFiles = new System.Collections.ArrayList();

// Loop to recieve all data sent from client
while ((i = stream.Read(bytes, 0, bytes.Length)) != 0)
{
// Clear buffers again
data = null;
send = null;
string message = “OK”;
// Get data as string
data = System.Text.Encoding.ASCII.GetString(bytes, 0, i);
Console.WriteLine(“FIZZ_RCV: {0}”, data);

String[] command = data.Split(‘ ‘);

// Insert Possible Commands Here
if (command[0] == “FIZZ_ADDFILE”)
{
FizzSrvLight.Structure.File file = new FizzSrvLight.Structure.File(command[1], command[2], command[3], command[4], command[5], command[6]);
CurrentFiles.Add(file);
}
else if (command[0] == “FIZZ_RMVFILE”)
{
FizzSrvLight.Structure.File file = new FizzSrvLight.Structure.File(command[1], command[2], command[3], command[4], command[5], command[6]);
CurrentFiles.Remove(file);
}
else if (command[0] == “FIZZ_AUTH”)
{
string username = command[1];
string password = command[2];
}
else
{
Console.WriteLine(“FIZZ_INVALID_INPUT”);
Console.WriteLine(“Error Handled”);
message = “ERROR”;
}

send = message;

byte[] msg = System.Text.Encoding.ASCII.GetBytes(send);

// Send back an OK response;
stream.Write(msg, 0, msg.Length);
Console.WriteLine(“FIZZ_SND: ” + message);
}
System.Threading.Thread.Sleep(1000);
}
}
}
}

Now, time for the struct.

namespace FizzSrvLight
{
namespace Structure
{
public struct File
{
public string FileName;
public string FilePath;
public string FileType;
public string SharedBy;
public string IPAddress;
public string Blacklist;

public File(string name, string path, string type, string user, string ipaddr, string blacklisted)
{
FileName = name;
FilePath = path;
FileType = type;
SharedBy = user;
IPAddress = ipaddr;
Blacklist = blacklisted;
}

}
}
}

Well, there you have it. A very simple TcpListener Serve. Obviously theres better ways to do it but this is pretty simple, straight forward, and just all around easy. Please leave comments if you find bugs in it or see errors or even if you just don’t understand what some of it does.

Thoughts About Distributed Computing

For me, distributed and parallel computing is one of the most interesting areas in computer science that I have found so far. I would really love to do work/research with it.

Distributed and Parallel Computing, at least from how I see it, seems to be the future of computers and processing power. Processors are starting to reach towards their physical limitations. So instead of making one processor even better, why not just use several, or even hundreds.

Slowly, I’m putting together my spare parts that I have at home into more computers to use. I got two more working in the last 2 days, both of which are semi decent machines. The main thing that I want to do with them is to try to make some of my own distributed applications that will be able to spread its processes out over several computers. Right now I have 3 computers that I want to use in my distributed system, the most powerful of which will be the main computer that is in charge of assigning the tasks. the other two, which are not as powerful, but still can handle plenty, will be sent instructions that will be processed and then they will send results back to the main server.

Now seeing as the 2 secondary nodes that will be in the system do not have the same specs, I have to make sure the server pays attention to the % of their resources that are being used when it goes to assign instructions. If the slower computer has been assigned less, but is using more of its resources, the next task that needs to be processed will be sent to the other computer.

Currently both of my secondary nodes are running Windows XP. This is fine I guess, but it seems like a large waste of system resources to have to run full windows. Though I realize that there is no way in hell that I can actually do this, I would love to write my own very basic OS just to handle being a secondary node in a distributed system. A node that when it turns on, simply connects to the central server, waits for its tasks, and then executes and sends back reports. No need for hogging resources with a gui or anything. All that is particularly necessary is enough to execute instructions and send/receive data over the network. But, when you actually think about it, that would take a very long time for me to figure out how to do. I don’t know the first thing about where an operating system starts really. Maybe I could use a linux kernel and just build off of that and get rid of things that aren’t really needed. I need to do a lot of research on this one.

This seems like it has the opportunity to be extremely efficient. Just a lightweight OS meant solely for being a secondary machine in a distributed  system would be so much faster than running anything i could create as a windows application. Unfortunately, I don’t know where to begin. Maybe I will set this one aside as a very long term project, slowly do research on it, and spend the bulk of my time on other projects are already have going like Fizzure. Haven’t worked on that one in a few days. Need to get back to it. I’ve been doing some little practice applications with sending and receiving data with the TcpClient and TcpListener class in C#. Now that I think I have a much better handle on how those work, it should be much easier to get over the hurdle I was stuck on with Fizzure.

Another goal I have is to make the Fizzure central server able to be split into nodes. Have different sections of all the XML data to be searched stored on different nodes. when a query comes in, the server sends a request to each node, each node searches the part of the data that it has stored and returns its results to the server, which than returns all of the results to the client. This seems more efficient than just having it all done by one computer, though because of network bandwidth, I don’t quite know if it would be in actuality.

Ever Have Trouble Sending Things Between Your Home Computers? … Me Too

If you’re at all like me, I always have my friends bringing their computers over. We like to share stuff obviously, and although I always come up with some solution to the problem of “How do I get this file from my computer to yours?”, it’s never really the best answer, and generally involves a lot more work than it really needs to.

Last night, I was having a little LAN Party with 2 of my friends, and we were just hanging out playing all sorts of games like we do occasionally. But then of course one of my friends says, “Hey Ryan, lemme give you this file.”. Alright…too bad we don’t have a way for you to send it to me.

First try: “I’ll share a folder over the network and you can just drop it in there!” – FAILURE – Computers not in the same windows work group…

Second try: “Get on AIM, I’ll just send it to you!” – FAILURE – AIM sucks, and for whatever reason, I can never use it to send or receive files from any of the people that it matters for.

Third try: “Oh hold on I’ll put it on my USB Drive!” – FAILURE – “F***! I can’t find it. “

At this point, we are all angry. Suddenly I was struck with a thought. “Why don’t I just make my own program for LAN file transfers?”. Brilliance. Next came, “Why did it take me so long to think of that…”

So I spent a good bit of last night programming the night away like I generally do. I decided to write a simple thing in C#. Its nothing complex but it almost works. I’m having trouble making it so you can send files over about 50 MB though, which is a problem…

I get an exception on this line:


i = stream.Read(bytes, 0, bytes.Length);


It gives me an IOException and says:

Unable to read data from the transport connection: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

I’m not really sure what to do about that. I’ve tried fiddling with a few things, but it didn’t do anything. I’m going to ask one of my friends who is a professional developer about it when i talk to him next. If anyone else has suggestions, though, I would really appreciate them.

So thats pretty much what I’ve done recently. Nothing huge, but hopefully once I get past this error it will be done. I’ll let ya know once I get it completely working. If someone would like to download it or use it, I will prolly package it and put it on one of my servers and post a link.