It is my understanding that there is something called a blockchain that I have a very rudimentary understanding of. Blockchain seems to be like a way to keep record of transactions using Bitcoin or whichever cryptocurrency.

It's a chain of blocks. The blocks are records of transactions. New transactions get bundled up and assembled into a block. It's "chain-like" because, once a block is generally accepted, it takes extremely expensive computational effort to dislodge it. (Basically, you have to provide a plausible alternative chain that can replace the accepted one, and that's very hard to do).

Firstly, the way that blockchain is able to keep track of transactions but still remain anonymous.

Be careful with this. BitCoin is pseudo-anonymous. As critics have pointed out, the moment you send 0.0001 from your account to someone else, that person can now link you to all the other accounts you sent to. Granted, they don't have your name, or the name of the owners of those other accounts - yet. If they can link one of those payments to an exchange, they can snoop on (or subpoena) the exchange to find out the identity of the person who asked for their deposit to be sent to their address. Then they've got you. If the government really wants this information, they can track IP addresses or even network switches. They have some tricks up their sleeves, too. tl;dr don't assume perfect anonymity.

Also, how does the issue of a piece of cryptocurrency being used more than once? Is there a system in place that can prevent these same pieces from being used more than once? Double-spending?

They call that the double-spend problem. Google your cryptocurrency of choice and double-spend for a mathy explanation of the details.

At the same time that the government is projecting these sentiments about cryptocurrency, they are also holding a large amount of cryptocurrency,

My guess is that the government does have quite a few cloaked fingers deep into cryptocurrencies. It's not because I'm a conspiracy theorist. It's really very simple; a LOT of the top cryptographic talent either worked directly for, or had their research funded by, the government. You could compile a dream team of cryptocurrency developers from the NSA, for example. I don't know what the ultimate truth is, but I certainly wouldn't count them out, or assume they've been outsmarted.