1. Post #1
    Gold Member
    Smartguy5000's Avatar
    August 2008
    702 Posts
    I am currently working on creating a cluster computer for my school. I'd like to know what you guys think would be a good distro to install on the nodes and on the host.

    I was thinking Pelican http://idea.uab.es/mcreel/PelicanHPC/

    I'd like to use it for either car crash physics simulations, or if that doesn't pan out, as a render farm for CGI.

    So far, I have only the gigabit switch, and gigabit ethernet cables, I need to get the machines donated from local businesses, and possibly our school district's property redistribution warehouse. Hoping for between 20-47 nodes. I'll keep the thread updated with pictures when the project begins to move forward, and I'm sure I'll have more questions...

    So facepunch linux gents, what do you think?
    Reply With Quote Edit / Delete Reply Windows 7 United States Show Events Friendly Friendly x 1Funny Funny x 1 (list)

  2. Post #2
    Gold Member
    PvtCupcakes's Avatar
    May 2008
    10,900 Posts
    I think Dragonfly BSD is used for that. I know it's not Linux, but close enough.

    I'm sure Scientific Linux is also pretty good at it.

  3. Post #3
    Brock Obama's Avatar
    January 2012
    412 Posts
    You should call up Pixar and ask them what they use for their rendering farms.
    Reply With Quote Edit / Delete Reply Windows 7 United States Show Events Funny Funny x 2Late Late x 1 (list)

  4. Post #4
    Gold Member
    JohnFisher89's Avatar
    August 2007
    2,181 Posts
    For clustering like CGI you'll need FC or external SAS to provide the throughput needed. good luck though

  5. Post #5
    I survived Camp FP 2010
    Mattz333's Avatar
    June 2007
    1,662 Posts
    I has to write an essay on this last semester, i found beowulf clusters pretty interesting. Give the openMOSIX project a read, apartently it's pretty simple to set up.

  6. Post #6
    Lyoko2's Avatar
    September 2007
    386 Posts
    Cluster computing is really useful, anyone knows how can i setup my own cluster computing system in my home? i hope it will be easy.

  7. Post #7
    Brock Obama's Avatar
    January 2012
    412 Posts
    Cluster computing is really useful, anyone knows how can i setup my own cluster computing system in my home? i hope it will be easy.
    Read the post above yours.
    Reply With Quote Edit / Delete Reply Windows 8 United States Show Events Zing Zing x 1 (list)

  8. Post #8
    Lyoko2's Avatar
    September 2007
    386 Posts
    Read the post above yours.
    yeah but i noticed these tools require me to compile stuff, and even more programming,
    i would like a Ready to use solution..

  9. Post #9
    Brock Obama's Avatar
    January 2012
    412 Posts
    yeah but i noticed these tools require me to compile stuff, and even more programming,
    i would like a Ready to use solution..
    I personally don't know of any, since I've never built a cluster. Hopefully somebody can point you in the right direction soon. :)
    Reply With Quote Edit / Delete Reply Windows 8 United States Show Events Agree Agree x 1 (list)

  10. Post #10
    Gold Member
    Smartguy5000's Avatar
    August 2008
    702 Posts
    So far the trickiest part is getting the hardware I need. I've got to talk to several different companies.

    Edited:

    For clustering like CGI you'll need FC or external SAS to provide the throughput needed. good luck though
    Gigabit isn't fast enough?
    Reply With Quote Edit / Delete Reply Windows 7 United States Show Events Agree Agree x 1 (list)

  11. Post #11
    I wasted a dollar on a stupid title.
    nikomo's Avatar
    September 2007
    17,101 Posts
    yeah but i noticed these tools require me to compile stuff, and even more programming,
    i would like a Ready to use solution..
    ...

    What do you want a cluster for? I honestly doubt you have an actual need for clustering.
    Reply With Quote Edit / Delete Reply Windows 7 Finland Show Events Agree Agree x 1 (list)

  12. Post #12
    Gold Member
    Mr.T's Avatar
    August 2010
    2,691 Posts
    ...

    What do you want a cluster for? I honestly doubt you have an actual need for clustering.
    Rendering. I plan on building a Raspberry Pi cluster or a cheap computer cluster for rendering Blender.
    Reply With Quote Edit / Delete Reply Windows 7 Sweden Show Events Funny Funny x 7Dumb Dumb x 4Winner Winner x 1 (list)

  13. Post #13
    Gold Member
    JohnFisher89's Avatar
    August 2007
    2,181 Posts
    Edited:



    Gigabit isn't fast enough?
    125MB/s is slow

    10Gig is better throughput for clustering
    Fibre Channel if you want no loss, ultra performance, TCP/IP is shit for clustering

    But for home stuff I guess who cares 125MB/s assuming the switch has the processing power to handle <n_ports> X 125MB/s throughput
    Reply With Quote Edit / Delete Reply Windows 7 United States Show Events Agree Agree x 1 (list)

  14. Post #14
    RUBY OVERLORD
    swift and shift's Avatar
    November 2011
    2,115 Posts
    have a look at baremetal node

    Edited:

    http://www.returninfinity.com/baremetalnode.html
    Reply With Quote Edit / Delete Reply Mac Australia Show Events Agree Agree x 1 (list)

  15. Post #15
    Gold Member
    Smartguy5000's Avatar
    August 2008
    702 Posts
    125MB/s is slow

    10Gig is better throughput for clustering
    Fibre Channel if you want no loss, ultra performance, TCP/IP is shit for clustering

    But for home stuff I guess who cares 125MB/s assuming the switch has the processing power to handle <n_ports> X 125MB/s throughput
    We've got a shitton of RJ45 connectors, and I might be able to get funding for a roll of Cat6A and a handfull of NICs that support 10GB BaseT
    Reply With Quote Edit / Delete Reply Windows 7 United States Show Events Winner Winner x 1 (list)

  16. Post #16
    Gold Member
    JohnFisher89's Avatar
    August 2007
    2,181 Posts
    We've got a shitton of RJ45 connectors, and I might be able to get funding for a roll of Cat6A and a handfull of NICs that support 10GB BaseT
    nice are the switches 10GB?

    if the nics support 10GbE see if they support FCoE, if they do you'll get a good speed boost

  17. Post #17
    I wasted a dollar on a stupid title.
    nikomo's Avatar
    September 2007
    17,101 Posts
    Rendering. I plan on building a Raspberry Pi cluster or a cheap computer cluster for rendering Blender.
    You can build a 300$ rig that will render shit twice as fast as 600$ worth of Raspberry Pi's.

    Not to mention, due to 1 per customer limitations on first batch, you wouldn't be able to get enough of them for a cluster for ages.
    Reply With Quote Edit / Delete Reply Windows 7 Finland Show Events Agree Agree x 6Dumb Dumb x 1 (list)

  18. Post #18
    Follow me on github!
    benjojo's Avatar
    January 2009
    2,622 Posts
    I can back this up, I've had great communication from the author.
    Reply With Quote Edit / Delete Reply Windows 7 United Kingdom Show Events Agree Agree x 1 (list)

  19. Post #19
    Gold Member
    Smartguy5000's Avatar
    August 2008
    702 Posts
    nice are the switches 10GB?

    if the nics support 10GbE see if they support FCoE, if they do you'll get a good speed boost
    I hope so, I'm trying to get it donated by cisco.
    Reply With Quote Edit / Delete Reply Windows 7 United States Show Events Funny Funny x 2 (list)

  20. Post #20
    pkt
    pkt's Avatar
    May 2008
    577 Posts
    If you plan on using Blender then I can't imagine that you'll need much as far as networking goes. Check out the cluster builds on graphicall and maybe blender.org if they still support it officially since the 2.5 thing. I dunno much else. How it helps.

  21. Post #21
    Gold Member
    JohnFisher89's Avatar
    August 2007
    2,181 Posts
    I hope so, I'm trying to get it donated by cisco.
    Most Cisco equipment will do FCoE or at least iscsi w/ jumbo frames

  22. Post #22

    January 2012
    68 Posts
    I've seen these things being built for a physics lab, couldn't pester them with questions though - as they were the snooty Oracle consultant type.

    The OS doesn't really matter, but i would go with an enterise grade linux like RHEL. As cluster configs are prob well documented.

    ww.centos.org/docs/5/pdf/Cluster_Administration.pdf
    CentOS is zero cost RHEL

  23. Post #23
    Gold Member
    Smartguy5000's Avatar
    August 2008
    702 Posts
    Emailed a gentleman at a local university, and he said he'd forward my request for parts to their GRID processing professor, who recently oversaw upgrades of their system. Waiting to hear back from the 2nd guy.

  24. Post #24
    T3hGamerDK's Avatar
    January 2011
    2,551 Posts
    Wouldn't this cause trouble when you're going to run a lot of tasks?

  25. Post #25
    I wasted a dollar on a stupid title.
    nikomo's Avatar
    September 2007
    17,101 Posts
    Wouldn't this cause trouble when you're going to run a lot of tasks?
    Err, how?
    You create a task and then you deploy it to all the notes with the orchestrator and you're done.

    If you're planning on running a lot of tasks on the same cluster, you're doing it wrong.

  26. Post #26
    Gold Member
    Smartguy5000's Avatar
    August 2008
    702 Posts
    Still havent heard back, going to email them again.
    Reply With Quote Edit / Delete Reply United States Show Events Friendly Friendly x 1 (list)

  27. Post #27
    Gold Member
    Smartguy5000's Avatar
    August 2008
    702 Posts
    Ok, so finally an update. We'll be using CentOS.
    We got 47 nodes, IBM Xseries 330's with pentium 3's in them
    Some have 512mb of ram, and 10 are mirrornet capable with 1gb of ram
    We have gigabit ethernet switches that all connect back to a fiber switch
    I'll be posting pictures as soon as I get them!
    Heres one picture, I'm on the left. My best friend on the right.

    Reply With Quote Edit / Delete Reply Windows XP United States Show Events Winner Winner x 6 (list)

  28. Post #28
    2014 SH Pun Award Nominee
    Awesomecaek's Avatar
    January 2009
    21,482 Posts
    Noice.

    Get some liquid cooling and connect it to your heating system of your school, and you have heating for the school sorted out, as well.
    Reply With Quote Edit / Delete Reply Linux Czech Republic Show Events Funny Funny x 5 (list)

  29. Post #29
    T3hGamerDK's Avatar
    January 2011
    2,551 Posts
    Ok, so finally an update. We'll be using CentOS.
    We got 47 nodes, IBM Xseries 330's with pentium 3's in them
    Some have 512mb of ram, and 10 are mirrornet capable with 1gb of ram
    We have gigabit ethernet switches that all connect back to a fiber switch
    I'll be posting pictures as soon as I get them!
    Heres one picture, I'm on the left. My best friend on the right.

    You should run at least one Minecraft server on one of those, just for the kicks of it. (Since it doesn't take up a lot of space or RAM)
    Reply With Quote Edit / Delete Reply Mac Denmark Show Events Dumb Dumb x 10 (list)

  30. Post #30
    Gold Member
    Smartguy5000's Avatar
    August 2008
    702 Posts
    So yesterday I moved them into their actual home, the main patch closet at the school. Tomorrow I'm going to be mounting them in the racks, and attempting to get rocks to boot to pXe. Rocks is a CentOS based distro designed for HPCs

  31. Post #31
    >:C
    Chulo Arco's Avatar
    January 2011
    335 Posts
    OP, I officially love you. You've answered a question I've been looking for for ages.
    Reply With Quote Edit / Delete Reply Windows 7 Belgium Show Events Agree Agree x 1 (list)

  32. Post #32
    Raxas's Avatar
    December 2009
    2,594 Posts
    -snip-

  33. Post #33
    Gold Member
    Smartguy5000's Avatar
    August 2008
    702 Posts
    Well so much for mounting them in the racks, no side rails... Going to have to stack them for now, and get them functional to prove to administration that they can be a revenue stream before I can buy rails kits for then.

  34. Post #34
    Gold Member
    PvtCupcakes's Avatar
    May 2008
    10,900 Posts
    Minecraft server.. .doesn't take up a lot of space or RAM
    Lol
    Reply With Quote Edit / Delete Reply Linux United States Show Events Agree Agree x 6 (list)

  35. Post #35
    T3hGamerDK's Avatar
    January 2011
    2,551 Posts
    Compared to most other process and RAM intensive tasks, I don't think it's taking up that much. On my own VPS, running Craftbukkit-git with SpoutPlugin, it runs perfectly fine with just about 256MB virtualized, in an ArchLinux virtual machine with 512MB RAM (it's also running a few experiments of my own written in NodeJS and Meteor).

    I've had loads of other services require AT LEAST 512MB of RAM, while this seems to be enough for all of my stuff.

  36. Post #36
    Gold Member
    gparent's Avatar
    January 2005
    3,949 Posts
    Compared to most other process and RAM intensive tasks, I don't think it's taking up that much.
    My master nameserver runs bind9 (master), ntpd, fail2ban, rsyslog and cron, a irssi session in tmux, and odentd. It uses 92 MB of RAM.

    One of my nameservers runs bind9 (slave), ntpd, fail2ban, rsyslog and cron and uses 19 MB of RAM.

    Another server I have runs apache2, nginx, passenger, a ruby application, ntpd, mysqld, cron and rsyslog, exim4, a redis server and bugzilla (a perl app). It uses 287 MB of RAM.

    I'm inclined to agree with PvtCupcakes here.
    Reply With Quote Edit / Delete Reply Windows XP Show Events Informative Informative x 2 (list)

  37. Post #37
    T3hGamerDK's Avatar
    January 2011
    2,551 Posts
    My master nameserver runs bind9 (master), ntpd, fail2ban, rsyslog and cron, a irssi session in tmux, and odentd. It uses 92 MB of RAM.

    One of my nameservers runs bind9 (slave), ntpd, fail2ban, rsyslog and cron and uses 19 MB of RAM.

    Another server I have runs apache2, nginx, passenger, a ruby application, ntpd, mysqld, cron and rsyslog, exim4, a redis server and bugzilla (a perl app). It uses 287 MB of RAM.

    I'm inclined to agree with PvtCupcakes here.
    Really? What Linux system are you using? I'm not very efficient, as I'm using a standard Arch Linux installation without GUI.

  38. Post #38
    Gold Member
    gparent's Avatar
    January 2005
    3,949 Posts
    Really? What Linux system are you using? I'm not very efficient, as I'm using a standard Arch Linux installation without GUI.
    I'm using Debian 6.0.5 64-bit. The numbers for the 287MB server are a bit low, the server will bump up to 350/400 when the 2-4 apps start to spin up but that's about as far as it will go.

  39. Post #39
    T3hGamerDK's Avatar
    January 2011
    2,551 Posts
    I'm using Debian 6.0.5 64-bit. The numbers for the 287MB server are a bit low, the server will bump up to 350/400 when the 2-4 apps start to spin up but that's about as far as it will go.
    Oh, so those numbers are on idle? Oh, I thought you meant during load. But I don't know Debian used that little, thats pretty cool.
    After checking it out, I realize that my server does use a lot less than I had measured (I was counting the virtualization usage, instead of reading the usage directly from an SSH session using htop), but still not as low as yours. I should probably look into getting a server kernel too. Thanks for the info!

  40. Post #40
    Gold Member
    gparent's Avatar
    January 2005
    3,949 Posts
    Oh, so those numbers are on idle? Oh, I thought you meant during load. But I don't know Debian used that little, thats pretty cool.
    After checking it out, I realize that my server does use a lot less than I had measured (I was counting the virtualization usage, instead of reading the usage directly from an SSH session using htop), but still not as low as yours. I should probably look into getting a server kernel too. Thanks for the info!
    If your vps panel includes hard drive caching then yeah you have to discard that. You can use 'free' to get the numbers you need. My secondary dns and primary dns don't really rise up in memory, but like I mentioned in my last post, the app server does.
    Reply With Quote Edit / Delete Reply Windows 7 Show Events Useful Useful x 1 (list)