Oh gosh, I don't know that this is a question that can be answered or even have one definitive answer. Certianly folks will feel strongly about one game versus another, but remember when we had an e-mail recently on OLR where someone asked "what is most definitive game of the PS3/360/Wii generation" and there was an e-mail that was sent in challenging that saying that no matter what the answer will always be relative?
I think there's a temptation too to automatically nominate retro games because, well if it's older it must be more significant, right? but not necessairly JUST retro.
here's some games I would consider as the most significant of all time... both retro and modern:
Halo 2 - Gave rise to the Xbox Live platform and online console gaming which one could argue is responsible for the decline that we're seeing at Arcades now where it's becoming Las Vegas with tickets for children, as gamers no longer have to adventure out to the arcade to find people to play games with at a moment's notice.
Street Fighter II - For almost the opposite reason as the above, LOL, probably the single biggest arcade game in terms of competitive gaming and venturing out to the arcade to find new people to test skills against. Almost every gamer of a certain age or older has an epic Street Fighter II story to tell.
Tetris - Before there were Facebook games, there was Tetris. Before we teased Farmville and Candy Crush Saga of being soccer mom games, soccer moms were playing Tetris on the Game Boy. Tetris's big peak with the Game Boy release came at a time when there was a big focus on video game violence and warning labels. Tetris wasn't any of that, it was the ultimate game for filthy casuals
Tie: The Oregon Trail, Where in the World is Carmen Sandiego, Number Munchers. Chances are, if you are in your 30s, one of these games was your gateway drug to video games growing up. This was a chance to play video games in school and still get an A! They showed that games can help a person learn, or probably more rather re-enforce learning.
World Of Warcraft - Although I could easily credit Ultima Online or Everquest with this, I think World of Warcraft kind of perfected the blend of social interactions mixed with gaming. Obviously it's the most successful MMO ever (outside of Asia). Not only though is there good news with social interaction, questing with friends, forming real friendships and RELATIONSHIPS inside a video game, but there's also some of the not so great things it's done for gaming, like really expose video gaming as something you can get seriously addicted to and detach from the "real" world.
Space Invaders - I think a lot of what you credit Pong with can also be credited to Space Invaders. I think sometimes we forget just how freakin huge this game was-- especially in Japan. Space Invaders was like, the reason arcades were born. In an interview once, Miyamoto said that he wasn't interested in video games until he saw Space Invaders. Therefore, one could argue that without it, we wouldn't have even had Super Mario Bros in the first place!
That's just off the top of my head, I'm sure some of the fine VOG folk out there can think of even more...