1. Post #1
    Gold Member
    VoiDeD's Avatar
    August 2005
    840 Posts


    Hello all,

    I decided to make a thread to inform everyone about some recent developments that have happened with the SteamRE project.

    For those of you who haven't heard of SteamRE (and I assume that is a large majority), it is quite simply an ongoing effort to understand the physical networking behind Steam itself.

    The team behind it includes Chrisaster, stgn, ComWalk, and myself.

    Some of you may remember a few posts I've made in the Open Steamworks thread about the project.

    Needless to say, the project got very far initially and then, school started and all progress pretty much stopped. The SteamRE project didn't move anywhere until a few days ago when the "Midwest Blizzard of 2011" gave me a few days free time to get some work done.

    Today I want to unveil some fruits of all our efforts that the team has put in. We've tinkered with the protocol enough that we have a pretty clear understanding of how the protocol itself works, and we're able to achieve full log on, and friend messaging capabilities.

    We've developed a heavily WIP C# library called SteamKit2 that implements the networking, and allows developers the control to make their own clients. We've also developed our own POC client to test out code as we write it.

    The code is freely available at this Bitbucket proejct: https://bitbucket.org/VoiDeD/steamre.

    Keep in mind that this project is a WIP, and if you're going to use SteamKit2 in your projects, there will be breaking changes in the future. The library is nowhere near stable at this point, but we're going to continue working on it.

    [release]Here's a video that shows the proof of concept client in action: http://dl.dropbox.com/u/99606/osw_st...02-03_1720.swf

    A few things to note here, this is running on Ubuntu. This is an entirely stand alone client. I've said that before, but this time it's completely stand alone and doesn't require any libraries or dependencies on steam.

    Potentially you can use this client on any operating system that supports Mono.[/release]


    Edited:

    I forgot to mention some contact details.

    The primary place we use for discussing development is the opensteamworks IRC channel.

    You can find it on irc.gamesurge.org @ #opensteamworks.
    Reply With Quote Edit / Delete Reply United States Show Events Programming King x 93Winner x 5Useful x 4Friendly x 2Zing x 1Informative x 1Optimistic x 1 (list)

  2. Post #2
    Gold Member
    FoohyAB's Avatar
    July 2009
    901 Posts
    Holy shit, finally a way for me to easily talk with steam friends without having to start up steam itself.
    Reply With Quote Edit / Delete Reply United States Show Events Zing Zing x 1 (list)

  3. Post #3
    Moderator
    Novangel's Avatar
    September 2008
    19,009 Posts
    Now for an app
    But useful as shit anyway.
    Reply With Quote Edit / Delete Reply Australia Show Events Agree Agree x 3 (list)

  4. Post #4
    Gold Member
    foszor's Avatar
    April 2005
    1,038 Posts
    I would like to write an Android chat client, got any documentation?

    Edited:

    Btw this is awesome
    Reply With Quote Edit / Delete Reply United States Show Events Winner Winner x 31Programming King Programming King x 2 (list)

  5. Post #5
    Gold Member
    VoiDeD's Avatar
    August 2005
    840 Posts
    I would like to write an Android chat client, got any documentation?

    Edited:

    Btw this is awesome
    The documentation, at this point, is entirely the code we have available. We used to have a wiki run by stgn, but it's gone now. Your best bet is following the POC client code.

    Also, I'm not entirely sure how you'd get it running on Android, since the current library requires Mono.

  6. Post #6
    Gold Member
    florian's Avatar
    January 2005
    238 Posts
    The documentation, at this point, is entirely the code we have available. We used to have a wiki run by stgn, but it's gone now. Your best bet is following the POC client code.

    Also, I'm not entirely sure how you'd get it running on Android, since the current library requires Mono.
    I guess thats the reason he requested documentation, so he could translate your findings in to Java.

  7. Post #7
    Moderator
    Novangel's Avatar
    September 2008
    19,009 Posts
    Also would've thought reverse engineering was against TOS.
    Reply With Quote Edit / Delete Reply Australia Show Events Funny x 15Programming King x 3Optimistic x 3Zing x 1Dumb x 1 (list)

  8. Post #8
    mmavipc's Avatar
    February 2009
    812 Posts
    Isn't this thread more suitable for a certain Russian counter-strike source site?
    Reply With Quote Edit / Delete Reply United States Show Events Dumb Dumb x 44Funny Funny x 7Agree Agree x 4Programming King Programming King x 1 (list)

  9. Post #9
    Hates php
    high's Avatar
    May 2006
    2,377 Posts
    Ergh, that receive code is making me go crazy.

  10. Post #10
    Gold Member
    VoiDeD's Avatar
    August 2005
    840 Posts
    Ergh, that receive code is making me go crazy.
    You should have seen the UDP code in the old SteamKit. What a disaster that was...

    For some reference, Valve's Steam3 implementation in Steam makes use of UDP by default, but they practically recreated TCP's reliability on top of it.

    http://tracker.limetech.org/projects...pConnection.cs

    I'm not much of a networking guru, but it seems to me that you may as well have used TCP at that point.
    Reply With Quote Edit / Delete Reply United States Show Events Informative Informative x 1 (list)

  11. Post #11
    Gold Member
    Quark:'s Avatar
    January 2011
    5,777 Posts
    Hopefully I can put this on a USB to talk to my friends on steam when I'm away from my home computer. :-)
    Reply With Quote Edit / Delete Reply United States Show Events Agree Agree x 2 (list)

  12. Post #12
    ghettochip's Avatar
    February 2011
    91 Posts
    You should have seen the UDP code in the old SteamKit. What a disaster that was...
    old steamkit was a disaster in general. it wasn't even a library and passwords in steam3 were hardcoded at one point.
    Reply With Quote Edit / Delete Reply Canada Show Events Agree Agree x 1Funny Funny x 1 (list)

  13. Post #13
    Jaykin Bacon: Episode 3
    SteveUK's Avatar
    May 2005
    2,722 Posts
    since the current library requires Mono.
    Monodroid
    Reply With Quote Edit / Delete Reply United Kingdom Show Events Agree Agree x 1 (list)

  14. Post #14
    ghettochip's Avatar
    February 2011
    91 Posts
    Monodroid
    embedding mono yourself would probably be more worth it.

  15. Post #15
    Hates php
    high's Avatar
    May 2006
    2,377 Posts
    You should have seen the UDP code in the old SteamKit. What a disaster that was...

    For some reference, Valve's Steam3 implementation in Steam makes use of UDP by default, but they practically recreated TCP's reliability on top of it.

    http://tracker.limetech.org/projects...pConnection.cs

    I'm not much of a networking guru, but it seems to me that you may as well have used TCP at that point.
    Rewriting "NetLoop" because it is going to bug me until I do.

    Also why do you have DataStream? Why not just use BinaryReader? Only reason I can see why is because BinaryReader doesn't have a Read<Type>() but you can always add it as an extension or create a class that inherits from it and adds it.

  16. Post #16
    Gold Member
    VoiDeD's Avatar
    August 2005
    840 Posts
    Also why do you have DataStream? Why not just use BinaryReader? Only reason I can see why is because BinaryReader doesn't have a Read<Type>() but you can always add it as an extension or create a class that inherits from it and adds it.
    Gives me convenience of not having to create a MemoryStream and BinaryReader for every buffer I need to read out of, and handles swapping endianness.

  17. Post #17
    Not a Gold Member
    cheesywlz's Avatar
    December 2008
    555 Posts
    Any chance that an iPhone steam client could be made with this?

  18. Post #18
    ghettochip's Avatar
    February 2011
    91 Posts
    Any chance that an iPhone steam client could be made with this?
    monotouch, embed mono yourself, or port it to obj-c (eugh).

  19. Post #19
    Person
    geel9's Avatar
    June 2008
    9,148 Posts
    I would like to write an Android chat client, got any documentation?

    Edited:

    Btw this is awesome

    God dammit I was going to do that.
    Reply With Quote Edit / Delete Reply United States Show Events Dumb Dumb x 20Disagree Disagree x 4Agree Agree x 1 (list)

  20. Post #20
    ghettochip's Avatar
    February 2011
    91 Posts
    God dammit I was going to do that.
    hahahaha good luck with that.
    Reply With Quote Edit / Delete Reply Canada Show Events Agree Agree x 1 (list)

  21. Post #21
    Hates php
    high's Avatar
    May 2006
    2,377 Posts
    Gives me convenience of not having to create a MemoryStream and BinaryReader for every buffer I need to read out of, and handles swapping endianness.
    Can always create a constructor for BinaryReader that takes a byte[] :P. Although for endianness you would end up needing to create a class anyways.

    Anyways, I rewrote NetLoop. Personally I would create something so that it would be a bit cleaner. Like a class that handles receiving X bytes instead of the 2 loops. But I didn't want to change too much. This is more just for showing. Anyways I did test it and it does work(Worked the first time too :D).

    void NetLoop()
    {
        while (true)
        {
            lock (ConnLock)
            {
                if (!bConnected)
                    return;
            }
    
            MemoryStream buf;
            int read;
            byte[] tmp = new byte[1024];
    
            buf = new MemoryStream(8);
            read = 0;
            while (buf.Position < buf.Length && (read = sock.Receive(tmp, Math.Min(tmp.Length, (int)(buf.Length - buf.Position)), SocketFlags.None)) != 0)
                buf.Write(tmp, 0, read);
    
            if (read == 0)
            {
                DebugLog.WriteLine("TcpConnection", "Recv'd connection closed!");
                return;
            }
    
            DataStream ds = new DataStream(buf.ToArray());
    
            uint packetLen = ds.ReadUInt32();
            uint packetMagic = ds.ReadUInt32();
    
            if (packetMagic != TcpConnection.MAGIC)
            {
                DebugLog.WriteLine("TcpConnection", "RecvCompleted got a packet with invalid magic!");
                return;
            }
    
            buf = new MemoryStream(packetLen);
            read = 0;
            while (buf.Position < buf.Length && (read = sock.Receive(tmp, Math.Min(tmp.Length, (int)(buf.Length - buf.Position)), SocketFlags.None)) != 0)
                buf.Write(tmp, 0, read);
    
            if (read == 0)
            {
                DebugLog.WriteLine("TcpConnection", "Recv'd connection closed!");
                return;
            }
    
            byte[] bufa = buf.ToArray();
            if (NetFilter != null)
            {
                bufa = NetFilter.ProcessIncoming(bufa);
            }
    
            OnNetMsgReceived(new NetMsgEventArgs(bufa));
        }
    }
    Reply With Quote Edit / Delete Reply United States Show Events Programming King Programming King x 4Friendly Friendly x 1Winner Winner x 1 (list)

  22. Post #22
    Person
    geel9's Avatar
    June 2008
    9,148 Posts
    hahahaha good luck with that.
    What does this mean, exactly?
    Reply With Quote Edit / Delete Reply United States Show Events Dumb Dumb x 15 (list)

  23. Post #23
    Gold Member
    foszor's Avatar
    April 2005
    1,038 Posts
    The documentation, at this point, is entirely the code we have available. We used to have a wiki run by stgn, but it's gone now. Your best bet is following the POC client code.

    Also, I'm not entirely sure how you'd get it running on Android, since the current library requires Mono.
    I guess I was more interested in writing it from the ground up, without the use of your library. Not entirely sure it's possible, was just curious to try.

  24. Post #24
    Gold Member
    VoiDeD's Avatar
    August 2005
    840 Posts
    I guess I was more interested in writing it from the ground up, without the use of your library. Not entirely sure it's possible, was just curious to try.
    We're writing up some preliminary documentation that should explain the networking more than the code will.
    Reply With Quote Edit / Delete Reply United States Show Events Friendly Friendly x 3 (list)

  25. Post #25
    Gold Member
    foszor's Avatar
    April 2005
    1,038 Posts
    We're writing up some preliminary documentation that should explain the networking more than the code will.
    Awesome
    Reply With Quote Edit / Delete Reply United States Show Events Artistic Artistic x 1Friendly Friendly x 1 (list)

  26. Post #26
    Gold Member
    Matthew0505's Avatar
    January 2009
    3,170 Posts
    Isn't this thread more suitable for a certain Russian counter-strike source site?
    You'd still need to get a Steam account with the game bought to play on VAC-enabled servers, so not really

  27. Post #27
    Venice Queen's Avatar
    January 2011
    187 Posts
    What does this mean, exactly?
    Reply With Quote Edit / Delete Reply Australia Show Events Winner Winner x 20Friendly Friendly x 1Dumb Dumb x 1 (list)

  28. Post #28
    mmavipc's Avatar
    February 2009
    812 Posts
    You'd still need to get a Steam account with the game bought to play on VAC-enabled servers, so not really
    Couldn't someone just take this work, put in some effort, and make their own steam servers(give me boxes, I have no idea what I'm talking about here)
    Reply With Quote Edit / Delete Reply United States Show Events Dumb Dumb x 2 (list)

  29. Post #29
    Gold Member
    VoiDeD's Avatar
    August 2005
    840 Posts
    Couldn't someone just take this work, put in some effort, and make their own steam servers(give me boxes, I have no idea what I'm talking about here)
    That idea in itself is interesting, but honestly holds no merit. I don't condone or like the idea of Steam piracy, but in practice it isn't very hard to patch a few checks in Valve's client.

    We're writing up some preliminary documentation that should explain the networking more than the code will.
    Alright, we've gone through and explained the gist of the inner workings and some other things.
    You can view the docs -snip. There is also an easier to view version -snip-

    Keep in mind that you'll have to reference the code in order to understand some parts of this. Also, if anyone has any questions regarding anything please go ahead and mention them so we can clarify.

    Edit: The docs have moved to http://tracker.limetech.org/projects/steamre/wiki/
    Reply With Quote Edit / Delete Reply United States Show Events Useful Useful x 1 (list)

  30. Post #30
    Gold Member
    Matthew0505's Avatar
    January 2009
    3,170 Posts
    Couldn't someone just take this work, put in some effort, and make their own steam servers(give me boxes, I have no idea what I'm talking about here)
    Valve doesn't care because the pirates can't play on Steam servers, plus it's already been done
    Reply With Quote Edit / Delete Reply Australia Show Events Agree Agree x 2 (list)

  31. Post #31
    Gold Member
    JustExtreme's Avatar
    April 2007
    7,105 Posts
    This is pretty impressive - as others have said in this thread, I'll find this handy for chatting to Steam friends when not on Steam.

    Thank you for your work guys, hope you have been finding it interesting.

  32. Post #32
    Gold Member
    jA_cOp's Avatar
    May 2006
    2,795 Posts
    You should have seen the UDP code in the old SteamKit. What a disaster that was...

    For some reference, Valve's Steam3 implementation in Steam makes use of UDP by default, but they practically recreated TCP's reliability on top of it.

    http://tracker.limetech.org/projects...pConnection.cs

    I'm not much of a networking guru, but it seems to me that you may as well have used TCP at that point.
    It's actually very common to implement TCP-like stream protocols over UDP for games. When it comes down to it, TCP isn't very well suited for most modern games at all. The biggest problem is that TCP doesn't give you fine grained control over data priorities, since all data is assumed top priority. TCP will make sure that all data reaches its destination, no matter how much time passes, and will also make sure that data is received in the order it was sent. In data-intensive real-time games, like FPS games etc, neither of these guarantees are very important, hence UDP is used as the base for a custom protocol, often coupled with a separate TCP connection for stuff like chat and transfer of large data chunks (like game resources).
    Reply With Quote Edit / Delete Reply Norway Show Events Agree Agree x 3 (list)

  33. Post #33
    Gold Member
    AzuiSleet's Avatar
    September 2007
    764 Posts
    It's actually very common to implement TCP-like stream protocols over UDP for games. When it comes down to it, TCP isn't very well suited for most modern games at all. The biggest problem is that TCP doesn't give you fine grained control over data priorities, since all data is assumed top priority. TCP will make sure that all data reaches its destination, no matter how much time passes, and will also make sure that data is received in the order it was sent. In data-intensive real-time games, like FPS games etc, neither of these guarantees are very important, hence UDP is used as the base for a custom protocol, often coupled with a separate TCP connection for stuff like chat and transfer of large data chunks (like game resources).
    Except this is the case of a service where reliability is important, none of the packets are supposed to be lost. TCP implements all the features they would need like flow and congestion control that would help when dealing with slower connections. There's no latency or priority requirement for a service like Steam unlike a real-time game server. Steam2 already runs on TCP, so we're not quite sure why UDP is used, perhaps it's how they're managing 2,500,000 concurrent connections?
    Reply With Quote Edit / Delete Reply United States Show Events Artistic Artistic x 1 (list)

  34. Post #34
    Gold Member
    jA_cOp's Avatar
    May 2006
    2,795 Posts
    There's no latency or priority requirement for a service like Steam unlike a real-time game server.
    Aha, yes indeed, I was thinking about the Source engine all along, completely unrelated :doh:

    Steam2 already runs on TCP, so we're not quite sure why UDP is used, perhaps it's how they're managing 500,000 concurrent connections?
    Assuming they run Linux-based servers I don't see the connection count itself being much of a problem, but perhaps it's still related.

  35. Post #35
    PiXeN's Avatar
    April 2009
    955 Posts
    God dammit I was going to do that.
    store the passwords in plaintext on your server
    Reply With Quote Edit / Delete Reply France Show Events Funny Funny x 9Disagree Disagree x 1Dumb Dumb x 1 (list)

  36. Post #36
    haushippo's Avatar
    June 2010
    371 Posts
    store the passwords in plaintext on your server
    exclusive leaked pic of his app:
    Reply With Quote Edit / Delete Reply Canada Show Events Zing Zing x 39Funny Funny x 5Artistic Artistic x 1Winner Winner x 1 (list)

  37. Post #37
    ( ͡° ͜ʖ ͡°)
    Giraffen93's Avatar
    December 2006
    20,944 Posts
    This thing with the chat protocol, you can use raptr to talk with people on steam, so it's not really new now is it?
    Reply With Quote Edit / Delete Reply Sweden Show Events Dumb Dumb x 5Disagree Disagree x 1Useful Useful x 1 (list)

  38. Post #38
    Venice Queen's Avatar
    January 2011
    187 Posts
    Assuming they run Linux-based servers I don't see the connection count itself being much of a problem, but perhaps it's still related.
    Knowing Valve, they'd be doing dumb shit like using threaded TCP so 500,000 connections would pose a challenge.

  39. Post #39
    Gold Member
    foszor's Avatar
    April 2005
    1,038 Posts
    Keep in mind that you'll have to reference the code in order to understand some parts of this. Also, if anyone has any questions regarding anything please go ahead and mention them so we can clarify.
    Yes I have a question.

    How the fuck did you guys figure all this out? This is incredibly well documented for having started with nothing at all.
    Reply With Quote Edit / Delete Reply United States Show Events Agree Agree x 5 (list)

  40. Post #40
    Gold Member
    VoiDeD's Avatar
    August 2005
    840 Posts
    Yes I have a question.

    How the fuck did you guys figure all this out? This is incredibly well documented for having started with nothing at all.
    Everything starts from nothing. SteamRE is just the culmination of years of work and research done by us and others.

    Our Steam2 documentation can be traced back to work done by a team who developed a tool called the "Universal Content Launcher".

    You can see the similarities of the networking here: http://tools.assembla.com/svn/ucl/tr...ServerClients/.

    Their efforts can most likely be traced back to a person named "steamCooker" from cs.rin.ru who did a vast amount of research into the Steam2 servers and how they work. I don't know exactly who else to credit for the majority of the Steam2 networking being figured out, as the parts that are required for Steam2 logon were already well researched before we started work.

    The Steam3 documentation is a combination of two things:

    A very long time ago Valve accidentally released a debug build of their steamclient library which handles all Steam3 networking. Through this debug build we managed to extract structures and data definitions that helped pave the way to figuring out the UDP protocol.

    The rest of the information we have gathered is simply the work of long months of research, disassembly, and packet analysis. Knowledge of the API that is exposed by the steamclient library also helped us understand some of the design behind the networking.
    Reply With Quote Edit / Delete Reply United States Show Events Informative Informative x 6Programming King Programming King x 4Useful Useful x 1 (list)