I am happy to announce the launch of one of my first independent websites: paster
Paster is a very simple app for collaborative development, where you can paste your code and share it with anyone you would like. It supports syntax highlighting of over 40 languages, and I plan to add more as I can. It was written in PHP as you might be able to figure out after using it and was written in about a day and a half. So, go forth and paste!
Personally, I’ve always been interested in distributed computing. It’s one subject I’ve done a lot of reading on. After seeing a bunch of posts on various Google blogs, such as this one, I’ve decided I want to work on making my own little cloud. I know I don’t have 500,000+ computers to work with like Google, but I do have 5 (or so…I think I can scrap up a few more out of my extra parts).
I’m starting to plan out everything but this would consist of a “manager” computer, and then slave computers to do the actual processing. I’m going to be building a web server specifically for this purpose and using some undecided DBMS (I’m thinking about SQLite).
At least to start with, each request will only be handled by one computer, although I do plan to eventually spread the processing of one request over multiple computers. I’d like to start with smaller goals. The database however will store information in sort of a “striping” manner, where each database computer will have the same structure, data will be separated over multiple computers, but not mirrored. There will be an application inside the manager to layer on top of SQLite (or whatever I decide on) to manage the returned results from all the combined computers.
Again, I’m going to start small and start with a basic web server that serves basic HTML, and then move up and support a dynamic language (Most likely Python or PHP, not sure which yet though, or hell, maybe both). After I get all this working I will make it distributed between computers and reporting to its “manager”.
The only problem with this plan is that it seems like there would be a bottle neck on the “manager” computer. I guess eventually I’ll have to support several of those, and use a load balancer to switch traffic between them.
I haven’t decided what language I’m going to use yet, but I’m leaning towards Python. One might argue that something of this magnitude should be done in a language like C or C++, but the problem is I think those languages would make something like this too complicated, and would require far more code to do less. Personally I think python would be great for the job, however I’m still deciding because I might also want to use C#.
Will definitely keep updating as I work through the project. Feel free to make suggestions on how I should go about things.
I got kind of bored today, and wrote a pretty simple web crawler with python and it turned out to be less than 50 lines. It doesn’t store output, I’ll leave that up to anyone who wants to use the code, because, well, theres just too many ways to choose from. Right now you pass it a starting link as a parameter and it will crawl forever untill it runs out of links. But that is not a likely condition. So here ya go. Have fun. Feel free to ask questions
import sys
import re
import urllib2
import urlparse
tocrawl = set([sys.argv[1]])
crawled = set([])
keywordregex = re.compile('<meta\sname=["\']keywords["\']\scontent=["\'](.*?)["\']\s/>')
linkregex = re.compile('<a\s*href=[\'|"](.*?)[\'"].*?>')
while 1:
try:
crawling = tocrawl.pop()
print crawling
except KeyError:
raise StopIteration
url = urlparse.urlparse(crawling)
try:
response = urllib2.urlopen(crawling)
except:
continue
msg = response.read()
startPos = msg.find('<title>')
if startPos != -1:
endPos = msg.find('</title>', startPos+7)
if endPos != -1:
title = msg[startPos+7:endPos]
print title
keywordlist = keywordregex.findall(msg)
if len(keywordlist) > 0:
keywordlist = keywordlist[0]
keywordlist = keywordlist.split(", ")
print keywordlist
links = linkregex.findall(msg)
crawled.add(crawling)
for link in (links.pop(0) for _ in xrange(len(links))):
if link.startswith('/'):
link = 'http://' + url[1] + link
elif link.startswith('#'):
link = 'http://' + url[1] + url[2] + link
elif not link.startswith('http'):
link = 'http://' + url[1] + '/' + link
if link not in crawled:
tocrawl.add(link)
I was thinking the other day. How would I do something on the internet and make my activity completely untraceable. The simplest form of the answer was to scatter where the packets come from.
So the idea of the packet scatter network was born. Anyone could sign up and host a server and there would be a large list of servers that the program would use. what the client would do is grab all outgoing packets and scatter them amongst the servers in the list, attaching a forwarding address. when each packet gets to it a server it is then sent on to its final destination, but anyone trying to trace the packets would find anywhere from 2 to thousands of sources.
I have absolutely no idea if this is even vaugley possible. But its and idea. It seems like it would only work for UDP because those are not ordered and such like TCP. But I’m not sure. I’ll ponder on it more I guess. Just an idea.
Recently I came up with the idea to create my own little IRC Community. So I set to work. After several days and a very large amount of cursing, I finally got my IRC Server up and running. So then, I decided to buy a domain so that people could connect without having to use my IP address. Anyways. I set my original server up on a windows machine. Tonight, I spent another 6 hours setting up a second node in the network on my friends linux box. And then we had to go through the painstaking process of linking the servers. But now we have it all working together.
Now, the hardest part. Making the community. I only have 2 or 3 friends that frequent IRC Chat rooms. But I’m hoping maybe this post will draw some people in. For anyone who is interested in the new network please feel free to join us at irc.slushnet.com. Everyone is welcome!
When I woke up this morning, my first thought was “Sweet, going dumpster diving today”. After thinking about it for a little while longer, I started to get a little depressed about going, it seemed almost like a waste of time on a monday morning, there wouldn’t be anything good in the dumpsters!
Despite these thoughts, I called up Joe and we went anyways. Bright and early. 10 oclock.
Now, of course, thinking I wouldn’t find anything, I practically struck gold. After careful observation of the electronics dumpster, Joe and i managed to pull out two old Compaq servers. One 5U rack server, and one enormous tower server. Each has 5 hotswap bays which I am very excited about. Now of course, seeing as they were in the dump, neither are in working condition, but I do plan to change this. Me and Joe will be directing attention to them Wednesday evening to swap some parts here and there and see what will do.
I was very surprised to see that the tower still had all 5 hard drives in it. I believe each one is 4.5 Gb, but since the computer doesn’t turn on, I don’t know for sure. Thats just what the labels on the bays said. The drives could be bigger. We’ll see once i get it up and running. I plan on swapping out the motherboard and power supply with newer ones. and then i can use the RAID controller thats already in there and hook up all the drives. We’ll see if we can get it working.
As for the rack, I’m not quite sure what I can do with that one. It has no drives and seems to only take SCSI drives which I don’t have any of. But i have yet to fiddle with that as well.
Definitely will write more on what happens with these computers soon.
Heres pics:
http://s297.photobucket.com/albums/mm238/theanti9/?action=view¤t=IMG_3529.jpg
http://s297.photobucket.com/albums/mm238/theanti9/?action=view¤t=IMG_3530.jpg
http://s297.photobucket.com/albums/mm238/theanti9/?action=view¤t=IMG_3531.jpg
http://s297.photobucket.com/albums/mm238/theanti9/?action=view¤t=IMG_3532.jpg
Ok so despite lack of recent blog activity, I have been fairly busy doing things. Not too much on the programming side which is really why I haven’t been posting. But lots has happened on the hardware side.
My friend Joe found a steel rack enclosure back behind the movie theater (don’t ask me why it was behind a movie theater because I honestly don’t know), and so we lugged it over to my house and stuck my racks in it. That is, after getting all the racks working. We are still working on clustering the 3 racks together but we are running into some difficulties with that. We’ll figure it out eventually. Anyways, heres pictures!
Ah, the library grows once again. I either need less books, or a bigger shelf. Its even starting to bow from the weight. And its out of room. Damn….
Web Requests are amazingly easy in C# – All you need is this:
WebRequest myWebRequest = WebRequest.Create("http://www.google.com");
WebResponse myWebResponse = myWebRequest.GetResponse();
Stream ReceiveStream = myWebResponse.GetResponseStream();
StreamReader readStream = new StreamReader(ReceiveStream);
String Response = readStream.ReadToEnd();
Returns everything from wherever you request. So simple. Need I explain more?
ok so this is a little code snippet. Lets say I have 10 text files in a folder and I want to add my ASCII signature to the end of all of them. I can put them in an array, and use a Foreach loop to do it to all of them at once. Heres how:
public static void write(String[] files)
{
foreach (String s in files)
{
FileStream fs = new FileStream(s, FileMode.Open, FileAccess.Write);
String[] line = {" ..jjLLtt \r\n",
" GGffiiGGLL \r\n",
" iiDD ,,EE,, \r\n",
" ttGG EEii \r\n",
" ;;EE.. ;;EEtt \r\n",
" LLEEffGGEEtt \r\n",
" ;;;;..KK;; \r\n",
" ;;,, ;;KK.. ;;;; \r\n",
" ..KKDD ii,,;;EEtt KKEE \r\n",
" ..LLtt LLDDDDtt LLjj \r\n"};
for (int i = 0; i < line.Length; i++)
{
Byte[] buffer = System.Text.Encoding.ASCII.GetBytes(line[i]);
try
{
fs.Write(buffer, 0, line[i].Length);
}
catch (Exception e)
{
// Do Nothing
}
}
fs.Close();
}
Now unfortunately because it gets rid of all the tabs, you can see what it actually looks like, but you can see it here
The parameter for the write method is an array of strings. The strings in this case happen to be file names. So the method will take all the filenames into it, and for each one, open them in a filestream, and using the filestream for each one, will write the ASCII signature in the file, and then close the filestream.
