The Team Grows

As of yesterday, another programmer has joined the Fizzure project. This makes me very happy but at the same time, a little worried.

I am very happy about having another programmer to help me on this project for a bunch of reasons. It gives me someone to help me out when I am stuck on a bug. Someone to bounce my ideas off of and see what they think. Someone to do another portion of the programming. It is a huge help in a lot of ways.

However, at the same time, I am a little worried. I’ve never really worked with someone in a team on a large project before. I have to figure out the best way to keep code and things organized well between both people.

I am thinking of using Mercurial to help manage the code.  Luckily, the project is easily split into 2 quite separate parts – the Server and the Client. I will be working on the server, and the other programmer will be working on the client. Both of us wanted to almost start over from a code perspective because what I had written was pretty messy. I think it was quite a good idea. I’ve looked at his basic re-design of the client, and it looks pretty good. We are hoping to implement a skinning system for the client so people can make the program look according to their own likes.

I was not happy with the way the server was organized before so I decided to completely redo the structure. The server is going to be a distributed system so that it will be handled by several computers instead of just overloading one. I’m trying to make it an easily scalable and efficient distributed system. We’ll see how it turns out.

I haven’t spent too much time working with the new addition to the team, but the time we have spent has been great. We discuss the best way to do the current thing that we are working on instead of just going right to the first idea that pops into our heads. This, in the end, I think will cause a much more efficient and well organized program. We’ll see how everything turns out.

Need Distributive Task Ideas!

Alright, so as I have stated before, Distributed Computing (DC) is one of my biggest areas of interest. I am starting to explore and experiment with it now that I have started to get the hang of network programming.

Unfortunately,  I can’t think of any tasks that are worthy of being distributed that I can experiment with! Boo~

I mean I have one idea. I program that finds prime numbers and looks for patterns in distance between each prime number and the prime number before it, and maybe even drawing a 3D graph of it if i can figure out how to do that. Prolly just a 2D graph…

But anyways, I just cant think of things that would really require intense processing power. Technically it doesn’t have to require THAT much, just enough that it would be faster to do on a few computers instead of one. Please leave me any ideas you might have!

Thoughts About Distributed Computing

For me, distributed and parallel computing is one of the most interesting areas in computer science that I have found so far. I would really love to do work/research with it.

Distributed and Parallel Computing, at least from how I see it, seems to be the future of computers and processing power. Processors are starting to reach towards their physical limitations. So instead of making one processor even better, why not just use several, or even hundreds.

Slowly, I’m putting together my spare parts that I have at home into more computers to use. I got two more working in the last 2 days, both of which are semi decent machines. The main thing that I want to do with them is to try to make some of my own distributed applications that will be able to spread its processes out over several computers. Right now I have 3 computers that I want to use in my distributed system, the most powerful of which will be the main computer that is in charge of assigning the tasks. the other two, which are not as powerful, but still can handle plenty, will be sent instructions that will be processed and then they will send results back to the main server.

Now seeing as the 2 secondary nodes that will be in the system do not have the same specs, I have to make sure the server pays attention to the % of their resources that are being used when it goes to assign instructions. If the slower computer has been assigned less, but is using more of its resources, the next task that needs to be processed will be sent to the other computer.

Currently both of my secondary nodes are running Windows XP. This is fine I guess, but it seems like a large waste of system resources to have to run full windows. Though I realize that there is no way in hell that I can actually do this, I would love to write my own very basic OS just to handle being a secondary node in a distributed system. A node that when it turns on, simply connects to the central server, waits for its tasks, and then executes and sends back reports. No need for hogging resources with a gui or anything. All that is particularly necessary is enough to execute instructions and send/receive data over the network. But, when you actually think about it, that would take a very long time for me to figure out how to do. I don’t know the first thing about where an operating system starts really. Maybe I could use a linux kernel and just build off of that and get rid of things that aren’t really needed. I need to do a lot of research on this one.

This seems like it has the opportunity to be extremely efficient. Just a lightweight OS meant solely for being a secondary machine in a distributed  system would be so much faster than running anything i could create as a windows application. Unfortunately, I don’t know where to begin. Maybe I will set this one aside as a very long term project, slowly do research on it, and spend the bulk of my time on other projects are already have going like Fizzure. Haven’t worked on that one in a few days. Need to get back to it. I’ve been doing some little practice applications with sending and receiving data with the TcpClient and TcpListener class in C#. Now that I think I have a much better handle on how those work, it should be much easier to get over the hurdle I was stuck on with Fizzure.

Another goal I have is to make the Fizzure central server able to be split into nodes. Have different sections of all the XML data to be searched stored on different nodes. when a query comes in, the server sends a request to each node, each node searches the part of the data that it has stored and returns its results to the server, which than returns all of the results to the client. This seems more efficient than just having it all done by one computer, though because of network bandwidth, I don’t quite know if it would be in actuality.

Ever Have Trouble Sending Things Between Your Home Computers? … Me Too

If you’re at all like me, I always have my friends bringing their computers over. We like to share stuff obviously, and although I always come up with some solution to the problem of “How do I get this file from my computer to yours?”, it’s never really the best answer, and generally involves a lot more work than it really needs to.

Last night, I was having a little LAN Party with 2 of my friends, and we were just hanging out playing all sorts of games like we do occasionally. But then of course one of my friends says, “Hey Ryan, lemme give you this file.”. Alright…too bad we don’t have a way for you to send it to me.

First try: “I’ll share a folder over the network and you can just drop it in there!” – FAILURE – Computers not in the same windows work group…

Second try: “Get on AIM, I’ll just send it to you!” – FAILURE – AIM sucks, and for whatever reason, I can never use it to send or receive files from any of the people that it matters for.

Third try: “Oh hold on I’ll put it on my USB Drive!” – FAILURE – “F***! I can’t find it. “

At this point, we are all angry. Suddenly I was struck with a thought. “Why don’t I just make my own program for LAN file transfers?”. Brilliance. Next came, “Why did it take me so long to think of that…”

So I spent a good bit of last night programming the night away like I generally do. I decided to write a simple thing in C#. Its nothing complex but it almost works. I’m having trouble making it so you can send files over about 50 MB though, which is a problem…

I get an exception on this line:


i = stream.Read(bytes, 0, bytes.Length);


It gives me an IOException and says:

Unable to read data from the transport connection: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

I’m not really sure what to do about that. I’ve tried fiddling with a few things, but it didn’t do anything. I’m going to ask one of my friends who is a professional developer about it when i talk to him next. If anyone else has suggestions, though, I would really appreciate them.

So thats pretty much what I’ve done recently. Nothing huge, but hopefully once I get past this error it will be done. I’ll let ya know once I get it completely working. If someone would like to download it or use it, I will prolly package it and put it on one of my servers and post a link.

What Would I Do With A Million Dollars?

I saw a post yesterday about what someone would do with a million dollars. He said he would buy some crazy macs of course. The specs were amazing. Reading that post got me thinking though – What would I do with a million dollars?

After about a good 30 seconds or so, the answer was quite clear. I would make my own data center, duh!

And of course I would save several servers for my own private use for research on Distributed Computing. Because I think that is really interesting and would love to do work with it in the future.

Anyways, I immediately set out to find some hardware that I would use in my new Data Center.

The first thing that you would need obviously is for a Data Center would be a place to hold data. So I found this:
LACIE 300961 2TB Gigabit Ethernet Shared Hard Drive

Connects through a 10/100/1000 Ethernet port, and each has a 2TB capacity. VIA C3 800Mhz processor and 256MB of RAM.

Price Per Unit: $859.99

Quantity: 100

Total Cost: $85,999.00

After buying these i would now have $914,001.00 left to play with. Time for some processing power.

I put together a custom rack. Quite powerful in my opinion. Heres the specs:

Custom Rack Unit

CPU:
2 x Intel Xeon X5450 Harpertown 3.0GHz Quad-Core Processor

2MB L2 Cache

1333 Mhz FSB and 64 bit support

Motherboard:

TYAN S5397AG2NRF Dual 771 Extended ATX Server Motherboard

Dual 771 CPU Sockets

1600/1333/1066 Mhz FSB

16 x 240 pin DDR2 Memory Slots

1 PATA / 6 SATA II

3 x On-Board Gigabit LAN
RAM:

Kingston 2GB 240-pin DDR2 SDRAM PC-5300 Server Memory

2GB Capacity

PC-5300 Speed

Power Supply:

iStarUSA TC-350R2U 2 x 320w Server Power Supply

24 Pin Power Connector

2 x 350 Watts

2 Fans

Of course, to wrap it all up, I need a case…

Case:
iStarUSA D-200-PFS Black Steel 2U Server Case

1 External 5.25″ Drive Bay

1 External 3.5″ Drive Bay

2 Internal 3.5″ Drive Bays

3 Expansion Slots

2U Form Factor

2 x 80mm Fans

Price Per Unit: $3,875.82

Quantity: 100

Total Cost: $387,582.00

Ok, so now at this point, I have $526,419.00 left to spend. Haven’t even spent half yet.

What is the next thing that any Data Center needs? Internet of course. The data center is no good if no one can access it.

If I were to get 30/15Mbps service, It costs $239.99/month for a 2 year agreement. which comes out to be $5,760 for two years. leaving me with $520,659.00

Now that seems like a large amount of money, but if you really think about it. I would probably use the rest for an office/building, utilities, and repairs and maybe some employees. So now that I look at it, starting a Data Center would be hard because you end up spending more money on other things and less on actual computers. That’s disappointing.

I Own The Windows Registry

As of last night, I officially own the windows registry. JK. But I did figure out how to store and retrieve information from it with C#. Makes the development of Fizzure much easier. Before I was just going to use XML to store everything. But I was like,

 Hrm, Why not just use the Registry. It would be so much easier…

And as it turns out. It is haha. I’m not sure how much info I will actually store in there, but right now it just stores your Username and Password. I need to figure out how to store the password in, well, not plain text… If anyone knows  how to do that, please let me know :]

Boo for inefficient applications!

Everyone has used one. Mostly because everyone has used Internet Explorer. But everyone has gotten one of those programs that is just slow and a total resource hog. We all know and hate them. You feel even worse, though, when you realize you have created one yourself. I was pretty happy when I posted last because I had gotten a connection between the server and the client and everything was working pretty well.

Unfortunately, now I realize that, even though it does not use a whole lot of memory, the server uses 99% of my CPU. And I have a decent CPU as well. I mean, I know why it does it but i guess i have to figure out a way around it. Heres the code snippet that I’m pretty sure causes the problems:


while (true) {
if (server.Pending()) {
server.AcceptSocket();
Console.Out.WriteLine("Connection Accepted");
}
}

As you can see, the code just creates an infinite loop that checks for connections. Unfortunately this causes it to check about 2 billion times a second. which obviously…would use all of your processors resources. I’m trying to use a timer that checks every second, but I’m having trouble implementing it in a console application. Maybe I will change the server to a GUI application. Might be easier to do a lot of things. I guess I’ll have to ponder it for a while

EDIT: The problem has been fixed! Hooray! With a little bit of help from my good friend Windy, Just putting System.Threading.Thread.Sleep(1000); at the end brought the CPU usage down to 00% :] And it even brought down the memory usage a little! Hooray!

w00t – Great Success

I know for most professional programmers and really anyone else in general it isn’t much of an accomplishment, but for me this is – I got the basic TCP connection working between the Fizzure client and server.

The server is just a lightweight console application. So far all it really does is accept connections. But for now I guess thats good enough. Today, I need to work on the XML usage of both the client and the server, and then the interaction with XML between the two. For efficiency I think I will do a lot of caching data. But I guess I need to figure out how to do that as well. I’m going to have to do a little more planning and deciding before I do too much programming.

Also though, I need to work on the GUI for the client.  I’m thinking about making it kind of skinnable. Or at least it will have different like color themes you can choose from. But eventually, I want it to be skinnable. People do seem to like that, it makes you feel like your program is just that much more personalizable.

Fizzure Will Go Open Source – When It’s Ready That Is

I have decided that my peer to peer application, Fizzure, will be open source! Exciting, I know.

However, I don’t think that I will open the source until I have a working beta build. Right now its still in core development. And it has a long way to go. Only the client will be open source. The central server that all will keep the information organized will be private and closed source. Sorry fellas. Out of luck on that one.

Fizzure will be the first application that I will publish officially, so I’m not completely sure how to go about doing so. My friend suggested that since I’m opening the source I should put it under the GPL. Sounds like a good idea, but, how do I do this? If anyone has suggestions for a n00b like me on how to go about doing such that would be great. If not, I’m going to ask again later anyways, once it is closer to an actual release.

Great News For me – Less Work!

Despite last nights extreme anger over the fact that I thought I was going to redo the structure of my Fizzure application, I learned today that I most likely do not! And this simple fact makes me very happy. There is almost nothing I hate more than having to redo work.

I came into school this morning with some of my code in hand and went straight to my computer science room. My teacher (the ever brilliant Dianne Meskauskas) took a quick look at my code. As it turns out I had just made a small mistake of where I had placed the creation of one of my objects.

The error I kept getting was a NullObjectException when I tried to access an array. I had made the creation of the object inside of a timer, so each time the timer ticked, it was creating a new object. On the first time it ticks, it creates the object “o” of the OptionControl class and fills the array SharedDirectories. On the second tick, it recreates “o” and tries to access the now empty array SharedDirectories, thus giving me the NullObjectException

Here is the little code snippet :

private void timer1_Tick(object sender, EventArgs e) {
OptionControl o = new OptionControl();
if (count == 0) {
loadWhat.Text = "Finding Shared Directories...";
o.SharedDirectories = o.getSharedDirs();
} else if (count == 1) {
loadWhat.Text = "Scanning Directories...";
o.DoScan();
} else if (count == 2) {
loadWhat.Text = "Shared Directories Found!";
} else {
Dispose(false);
}
count += 1;
}


My mistake here is on the first line. This line needs to be taken out of there and made as a class variable instead of inside a method, that way it can be accessed from any of the methods and have the same data as well.