Overview of Programming Contests

I have been competing in various programming contests for the past 10 years. A lot, if you ask me. But, more importantly, I have competed in quite a few different kinds of contests. My adventure started with classic algorithms, and later I switched to optimization problems. Currently I compete mostly in machine learning contests (as a part-time job) and I also do some for-fun contests.

Considering that there aren’t too many people with such broad experience, I thought I’d write a (relatively) short summary of popular types of programming (algorithmic?) contests. This is not a complete list – instead I’m focusing on the ones that are most popular and (in my opinion) most useful.

The structure is type-oriented rather than site-oriented. This means that rather than flooding you with some random sites with an added description, I organized everything into categories and combined sites/contests with similar characteristics. Since some of the sites offer several types of competitions, I had to list them several times.

Ideally, if you’re confused or just curious about the whole world of competitive programming, this should give you a comprehensive overview of the subject.

Classic Algorithms

Sometimes also called binary problems or (derogatorily) recreational algorithms. You’re provided with the problem statement and your goal is to write a, usually, short program that reads the input, processes it and outputs the computed result. Everything according to the given problem statement. In most cases, you have to submit the source code of your program, which in turn is compiled on a remote server and run against some hidden data. The thing common to all contests mentioned below is that your program is always considered either fully correct or incorrect, and there’s nothing in between.

Classic algorithms contests are usually an entry-level point for competitive programming. The difficulty range can span from things that are absolutely trivial to problems that only the problem writer is able to solve. They are a great practical exercise for many beginner programmers, as they give small, bite-sized challenges. At higher levels, they require highly-developed concentration, great long-term memory, problem-solving skills and often deeply-specialized knowledge. Getting decent at those contests develops a lot of skills that are easily-transferable to other areas in computer science. Especially if you focus on developing raw skills, instead of spending your time on solving hundreds of problems and hoping that you’ll see a similar problem in the future.

Competitive Leagues

There are two major sites that host regular competitions. The first one is topcoder with its Single Round Matches (SRMs). The second site is Codeforces.

Both places are quite similar. Competitions are held a few times a week (links to upcoming contests: TC and CF). They last around 2 hours. Each format uses several (3 for TC, 5 for CF) original classic problems created specifically for that round (in general, this is always the case for every contest, but I wanted to make this clear). You submit only the source of the program and the program is executed remotely on the servers. Your solution, in order to be evaluated as correct, needs to produce correct output and run within specified time and memory constraints. The catch is that you learn that your solution is correct only at the end of the contest. When the round is finalized, you get your ranking updated (similar to ELO rating system) which quite accurately represents your current level. Whole user base is split into two different divisions (based on their current ranking) and each division gets a different problemset tailored towards their skill level. Also, after each contest, you can view submissions from other people, the editorial is published (which explains the correct solutions for the problems) and all of the problems are added to the practice area so that you can solve later the tasks that you were unable to solve during the contest. This makes them a perfect place for training.

Those were the similarities, so what’s different? The biggest downside of TC (apart from bad management, but that’s another story) is that it uses an archaic java applet for its competitions. Although that makes the first contest more complicated than it should be (the badly designed website doesn’t help), but in the long run it isn’t much worse than an “HTML5” UI. Another difference is that problems in each site have a slightly different focus. Tasks on TC are usually skewed towards problem solving, sometimes even resembling puzzles, while CF contains a lot of “filler” (another word for unimaginative) problems based on data structures, but this largely depends on the problem writer. Based on my experience, TC problems (on average) are slightly more interesting, but CF has more variety in the problems – mostly because they have more problems per round. Both competitions feature an opportunity to find bugs in other people’s codes (and get additional points if you’re successful at it), but CF designed it horribly and the best thing to do, is to ignore it entirely.

Annual Onsite Competitions

There are four big competitions: Google Code Jam, Facebook Hacker Cup, Yandex.Algorithm and TopCoder Open (algorithm track). All of them are quite similar. In order to qualify for them, you have to compete in a series of online qualifiers. Usually they are done in a knock-out fashion, reducing the number of competitors in each subsequent round. There are no restrictions on who can compete (well, unless you’re unlucky and your country is on the embargo list for USA). Rounds are short, between 90 minutes and 3 hours and feature only classic binary problems. If you manage to qualify, they will pay for your transportation and accommodation. They also give out prize money to winners, but for most people the most important thing you can win is either the trip itself, or landing a job in a top IT company.

The competitions have small differences between them (quality of problems, round structure, submission system), but there are two things that they all share: It’s really hard to qualify for any of them (it’s highly unlikely to qualify when you have less than 2 years of experience, even when you’re smart and dedicated). The other is that one guy won all of them in 2014. And considering that all of those contests have high variance, it’s an unbelievable feat.

Online Judges

There are lots of them: SPOJ, UVA, Timus to name a few. Usually, they mostly serve as an archive for problems from past ICPC competitions.

It’s worth mentioning that nowadays, almost all contest sites allow some sort of practicing. So in theory, almost all sites serve as an online judge. The reason why I put online judges in a separate category is that they don’t offer anything else.

The are very few reasons why you should ever be on such site: You’re at the point where you feel you’re very bad at some type of problems and you’re searching for some very tough specific problems (which shouldn’t ever happen unless you’re aiming for being top100 in the world) and you can’t find them anywhere else. You’ve participated in a contest, and the problems were uploaded to a judge site so you want to try ones that you haven’t “AC-ed” or maybe some alternative solutions. You’ve got a lazy teacher for your algorithm class and he hates his job.

Optimization Problems

The standard example of this is a Travelling Salesman Problem. Those problems are characterized by the fact that you get a different score depending on the quality of your solution. And they are (or at least they should be) designed so that it’s impossible to obtain a perfect score.

Contests with optimization problems usually last much longer, and thus focus on different set of skills than their classic counterpart. Things like mental stamina, time management or out-of-the-box problem solving skills. In general, optimization problems require you to be much more versatile, but you don’t have to excel in any particular area. They are also closer to doing actual research, so if you want to tie your career with something other than software development, it’s a good idea to try them.

Marathon Matches

Unfortunately, there aren’t too many places where you can hone your skills in this area. Topcoder have Marathon Matches, but they no longer host regular competitions – something they would do until a few years ago. Most matches held under the MM banner fall into Machine Learning category (described below), the exception is the annual TopCoder Open tournament that has a Marathon category. Both the onsite final and the qualification matches use optimization problems. There are some rumors that they want to go back to doing regular matches, but so far nothing has changed.

Fortunately enough, while there aren’t too many contests, topcoder has a big archive of past problems used in those contests. So if your sole goal is to be better at optimization problems, you can practice on old problems. The upside of this is that you have an access to all of the winning solutions, plus usually there’s a “Post Your Approach” thread in the appropriate forum for each of those past contests.

There are also other contests where you can encounter optimization problems, but unfortunately I don’t think anything stands even remotely close to MMs. Probably the most popular contest in that category is Al Zimmermann’s Programming Contest, but the problems there are rather shallow and aren’t too interesting.

Machine Learning

Sometimes mistakenly called data science. This is the place where the money is, because there’s a big demand for people who are specialized in ML. At least big compared to the demand for people specialized in classic and optimization problems.

ML contests require much more knowledge than the other types of contests, and in general they aren’t that entertaining. Still, they are currently the easiest way to get some hands-on experience in this field. If you need some motivation, remember that jobs requiring ML skills are some of the highest paid in the whole IT field.

Marathon Matches (again)

I could link to the same place I did above. Topcoder combined optimization and ML problems into a single category. Or to be precise, at some moment MMs category was expanded to include ML contests as well.

ML contests at topcoder come pretty much only in one form. Since whole topcoder is a crowdsourcing platform (plus community build around it), clients sometimes have problems that cannot be solved using “simple” software contests. In some of those cases, the problem they’re dealing with can be wrapped in an ML contest that features decent prize money for the best-performing solutions. Since ML contests are still MMs, the whole submission system works exactly the same way as in optimization problems. The only difference is that ML contests are usually not added to practice area.

Kaggle

Kaggle is a website that is pretty much build around ML contests. I’ll focus on differences between TC and Kaggle. The biggest difference is that in Kaggle you submit only the output of your solution instead of submitting the whole program and this has a lot of consequences. First of all, you get access to the entire data set and there are no limits on the languages/libraries you can use. There aren’t any time limits and you can even use a whole cluster to compute the result if you want (and can afford it). Since people don’t submit source codes, you can’t look at solutions at the end of contest. On the flip side, community is much more active and there are lots of “practice” contests where people share their solutions while the contests are still running. Contests on Kaggle also last much longer (usually 2 months for contests featuring a prize money).

Overall both sites serve a slightly different purpose and both of them have its pros and cons. Kaggle should be more friendly for people who are more interested in high-level knowledge rather than low-level inner mechanisms of how various ML techniques work, while TC is probably better for people with a strong algorithmic background. The other way of looking at it is that in Kaggle you (generally) use tools and in MMs you (generally) write your own.

24 hours of fun

15 years ago, the first Challenge24 was organized. I won’t go into the history of it, since I actually know very little about it. But I would describe this contest as 24-hour madness. It’s an annual team contest held in Budapest. You get around 15 problems. The scope of the problems is absolutely huge: classical, optimization, games over TCP/IP, computer vision, sound analysis, puzzles, and things that cannot really be described using words. You bring your own hardware for the contest and the whole competition is done offline. Honestly, this is the most entertaining contest I have ever attended. And I’m saying this, even though I had fever last time I participated.

Challenge24 inspired Deadline24, which in turn inspired Marathon24 (both held in Poland). They are somewhat simplified versions of CH24, but still a lot of fun. Instead of having lots of problems, they usually have 3 games and the rounds begin at the same time when the competition starts. For example, one of the past games was the classic asteroids game played simultaneously for 30 players.

Since there’s limited number of teams invited to the finals, each of those contests hold an online qualifier. Since those contests are way less popular than other annual competitions, it’s actually quite possible to qualify to one, without any significant background in algorithms (or even programming).

There is one additional annual contest worth mentioning that is somewhat similar to those 24-hours contests: Internet Problem Solving Contest. This contest has still a lot of tasks, although they have more resemblance towards the classic algorithms.

Others

There are two big websites that were left out. The first one is CodeChef that has two different types of monthly competitions. Both of them are classic algorithms competitions. The other site is HackerRank, which is a mix of an online judge and a place which hosts some one-time contests that can cover all types of contests. The reason why I left those out is that both are well-known for the low quality of problem statements and in general you should avoid those (there are few exceptions though), considering there’s an abundance of other contests.

If you’re still in high school, then the most important contest for you is International Olympiad in Informatics. Each country has its own national olympiad with its own set of rules. In many countries, performing well at the national level of the IOI, is the easiest and safest way to get into the university of your choice.

ACM ICPC is a “classic algorithms” contest and it has the longest history of all of the contests/sites mentioned here. It’s an annual team contest for students. Each team consists of 3 people from a single university. The main goal of ICPC is the popularization of programming worldwide. This is achieved with the use of a complex multi-level qualification system and (also complex) eligibility criteria. This enforces diversity during the world finals. As a trade-off, depending on where you live, it’s either surprisingly easy to qualify for the world finals, or next to impossible. Practicing exclusively towards ICPC is probably the worst way to invest your time in programming contests as it promotes the most narrow skill set, the only exception is when you live in one of those “lucky” regions.

Honorable mention goes to Hello World Open for creating biggest bullshit competition of the year with a very successful marketing plan. If you look carefully, you’ll see that they even added their link in wikipedia page just after the launch of the contest. I guess if you produced the #1 top grossing app on mobile, you know that stuff pretty well. Seriously, topcoder should learn from them.

Another honorable mention goes to Imagine Cup. I attended Imagine Cup four times. Twice as a competitor, once as a judge and one last time as a helper. Over the years I was able to see how it devolves from an innovative annual competition that gathers students from many different areas (start-ups, digital art, algorithms/AI) to a shitty PR machine for Microsoft’s products. Judges were often selected due to political reasons and were in no way qualified to be performing their job. And all of the categories that didn’t directly promote MS stuff were discontinued. The truth is, Imagine Cups were the best onsite events I ever attended. At the same time, nowadays I would never ever recommend participating in them to anyone.

Project Euler is a special kind of an online judge (with a heavy math focus), where you use programming as a tool for solving problems, instead of it being the point in itself. If you really want to do some programming problems without the time pressure, I suggest you do Project Euler instead of any of the already-mentioned “OJs”. Also, since it’s actually quite popular, it will be much easier to find some help online in case you’re stuck on one of the problems.

22 thoughts on “Overview of Programming Contests

  1. Maru

    Hi! Nice and complete list! I

    What do you think about “coding games” such as CheckiO, CodinGame, and HackerRank? Mostly these websites have “easy” problems (compared to topcoder) and I believe their main target is programmers who have not at all/little knowledge of algorithms.

    Do you think it is interesting to build a game that requires programming abilities in order to play it?

    I remember a website called NotPr0n, which required some general computer knowledge in order to pass the levels :)

    Reply
    1. psyho Post author

      What do you think about “coding games” such as CheckiO, CodinGame, and HackerRank? Mostly these websites have “easy” problems (compared to topcoder) and I believe their main target is programmers who have not at all/little knowledge of algorithms.

      I understand that you’re talking about AI contests? I don’t have experience with those sites (or in case of HR, with AI contests on that site), so I can’t really comment on them. On codeforces someone asked somewhat similar question and I replied “(…) One problem I often encountered in writing AI for contests, is that I don’t gain too much out of them. I believe that competing in MMs (optimization stuff) will actually help you more in writing good AI. At least, in the long run. The biggest downside is that “single-player games” are just not that fun :)” I don’t know what the actual target of those sites is, but I definitely think that writing AI should be a much more entertaining than classic/optimization problems.

      Do you think it is interesting to build a game that requires programming abilities in order to play it?

      Interesting yes, but it would be fun to a rather small group of people.

      I remember a website called NotPr0n, which required some general computer knowledge in order to pass the levels :)

      Yeah I remember it too, there were lots of such sites, but I think this one was the first.

      Reply
      1. http://www./

        Merci, Jean-Marie, j’envoie immédiatement le synopsis du « Goût du vert », si subtil, à mon éditrice ! Si elle me signe un contrat, vous devrais-je quelque chose pour l’idée du titre ?Amitiés confraternelles

        Reply
      2. http://www./

        Piccola osservazione a Paolo:Quando dici che “il virus si propaga tramite le cartelle condivise”….ma come fa? Riesce a veicolarsi semplicemente su una cartella condivisa, senza che nessuno esegua alcun eseguibile? allora sì che è da preoccuparsi…Da quel che ne sapevo io finora, un virus si propaga se qualcuno lo fa eseguire….qualcuno ha da dare qualche ragguaglio a proposito?ciao a tutti….

        Reply
    1. psyho Post author

      Software contest and fun in one sentence, are you sure? :)
      On a more serious note, I should probably add a disclaimer that this overview is about algorithmic contests only.

      Reply
      1. SCaffrey

        Maybe I didn’t get it right.
        But don’t you challenge competitive programming because of your love in solving problems and challenging yourself?

        Reply
        1. psyho Post author

          You did get it right.

          But, I don’t really enjoy programming itself. And, I don’t feel particularly challenged when I know that the biggest obstacle I have is lack of knowledge (in some specific area). That’s why I find machine learning less interesting than optimization stuff, even though it can solve more problems that humanity is currently facing. I’m getting bored way too easily, and I doubt that I would be able to work as a software developer for more than a month. Can’t really tell, since I never did :)

          Reply
  2. Bartek

    “(…)Especially if you focus on developing raw skills, instead of spending your time on solving hundreds of problems and hoping that you’ll see a similar problem in the future.”

    I wonder, what do you mean by devoloping raw skills ?

    Reply
    1. psyho Post author

      I try to make a distinction (even though it’s often very blurry) between skills and knowledge. Learning knowledge is easy and it’s always a matter of time. Learning skills is harder, but often they can be applied to many areas.

      What I meant here is that you can approach practicing in quite many different ways. Grinding a lots of problems is probably the worst them. For example, every time you make a bug in your code, you can try to understand why you did and how you could avoid in the future.

      Sorry if I’m being confusing. I had a plan to talk about this specific thing more in the future. Although, considering the tempo in which I’m putting new posts, it make take a while :)

      Reply
      1. Bartek

        Since it’s topic that is really interesting for me, I hope you’ll find the time soon, to post your thoughts about this :)

        Reply
  3. shashank

    hey psyho!!! i want to tell you…My life in college is like yours only…I am studying Computing Science for about 2 years..Many times i think that this is not for me….But sometimes when I code some algorithm , I just think it is for me…But you know, I think I am wasting my time in college but I have nobody to help me out in person…I want to have fun in life and enjoy as much as I can but I normally get depressed when I see on future years cause I am wasting enough time….Plz help me out with this situation…One thing forgot to tell you..Last Aug, I went to some college fest and there was some different type of coding contests..I did good on debugging problems and little bit on algorithms…Give me advice on getting better everyday…

    Reply
    1. psyho Post author

      Well, you’re from completely different country than I am, so please take my advice with a grain of salt.

      Frankly speaking, no one will help you in figuring out what you really want to do in your life, because they are busy figuring it out themselves. People drop out from the studies mostly because they have some ideas what they want to do in their life and studies are interfering with that. If you don’t have any (or you’re just not sure), you should probably take your studies lightly (i.e. put in as little time as you must) and use your free time to try as many things as you can. Just do stuff, and you’ll find your own way eventually.

      Not sure if that answered your questions :)

      Reply
  4. Hai

    Hi Psyho

    I am one of your fans on TC. Always admire your performance in various MMs :)

    One thing I am curious to know, is whether you participated in Kaggle and what is your ranking there. You are the top ML contestant on TopCoder, so I’d assume you should be among the top in Kaggle?

    Besides, on most of the recent contests on TopCoder, I noticed that you almost always use some kind of customised Random Forest (and win). Do you think Random Forest as superior to other ML algorithms? Or is it only suitable for those problems on TopCoder? Do you use it on Kaggles contest and how does it perform vs others on Kaggle?

    Thank you.

    Reply
    1. Psyho Post author

      Hey,

      My relation to Kaggle is quite complicated :)

      In the past, I considered competing there few times (only in less typical contest, since only in those I might have any edge over Kagglers), but personal experience was pretty bad, since admins messed up all of those contest. In general, in every contest that wasn’t your typical Classification/Regression task, I saw a lot of incompetence from the Kaggle’s admins. That’s one of the reasons why I’m staying away from the Kaggle.

      Nowadays, I compete in ML matches pretty much only for the prize money (and to gain some knowledge in case my endeavor with gamedev utterly fail, which I hope it won’t). I don’t really enjoy doing ML – pretty much the only things you do there is tweaking and trying different standard techniques. Having more experience just means that you are better at narrowing down the search space of good solutions. There’s very little room for creativity in ML when compared to optimization problems. And the fastest way of getting better in ML is reading stuff.

      So, having those things in mind, there’s very little incentive for me, to compete on Kaggle. The expected $/h (for everyone, even for people who are better than me) is near zero. And I’m far better off reading the winners’ solutions than implementing them by myself. The contests on Kaggle are really long, and the final rankings are either completely random (not enough data / it’s easy to get near-optimal model) or require huge amount of work (huge data sets / contest require very specialized knowledge or huge amounts of tweaking). You may have noticed that there are no people who won more contests in Kaggle. It’s because it’s just not possible and there’s no real point in doing so. In other words, Kaggle is a great source of knowledge, but competing there is (quite often) poorly invested time. And for me, it would be complete waste of time.

      Since TC requires us to write our own libraries, it’s harder to switch from one method to another. I chose to specialize in RFs since they’re probably the most robust method that is currently available. It’s almost always possible to convert your problem into a form where RF is applicable. It’s also true that TC has overabundance of contests where RF is probably the best technique (at least without using complex ensembles). Even though as I said, I don’t complete on Kaggle. I think you can guess, where did I get my idea from, to specialize in RFs :)

      Reply
      1. Anonymous

        Thanks a lot for taking time to answer my questions in such thorough details :) It is really interesting to see your view on Kaggle competition, and the insight about why you use RFs extensively on TopCoder :)
        I almost missed your reply because I was expecting a notification email when you reply (I did check on “Notify me of new posts by email” when posting). There was no email, but I decided to revisit the blog anyway and check if there is any answer from you, and there it is! Thanks again.
        Last but not least, all the best for your journey with gamedev! I wish you come out as a winner as you always does on TopCoder :)

        Reply
      2. dimkadimon

        My relation to Kaggle is similar. Even though I have a PhD in Machine Learning and I have a reasonable understanding of ML techniques, I totally suck at implementing them for real world problems. The only way I would compete in Kaggle’s ML competitions is to use off-the-shelf ML software, but that’s little fun and will give me no chance to win.

        I have tried a few non-ML contests on Kaggle and quite enjoyed them. Did you try any of the Santa problems? I found them more interesting, because they are more about optimization than pure ML.

        Reply
        1. Psyho Post author

          “The only way I would compete in Kaggle’s ML competitions is to use off-the-shelf ML software” – that’s how most of the people win there ;) Also, I don’t like calling libraries software.

          About Santa problems. Those are pretty shallow problems clearly designed in order to use sponsor’s software. Solving single test case is absolutely unfair for people without access to more computing people. And lastly, I don’t want to compete against my teammate for 24h contests (Marek Cygan).

          Reply
  5. G

    Hey Psyho,

    Thank you for the post. I’ve learned a lot from your previous TC codes, and am a big fan of yours.

    I have one stupid question that I would be greatly appreciated if you can comment. I am a programmer currently working in gaming industry for a little less than 2 years. Studying programming competitively gets me to think that this is the one for my future. It can either enlarge my insights about the computer science, or allow me to work for the top IT companies around the world(or maybe both). For that reason, I plan to quit my job, and concentrate on competitive programming for a couple of years – my goal is to accomplish red in TC or 2,200 in CF. How do you think about fully dedicating a couple of years in Problem Solving, and seeking for the next career move afterwards? Am I too naive and ignorant to even think about this ?

    From your big fan from South Korea.

    Reply

Leave a Reply

Your email address will not be published.