A portable server for around $85. Yes, if you told someone just that 10 years ago, you were put in an isolation cell in the worst-case scenario.
But today it is possible, and nobody will call you crazy with this setup you can put in tiny handbags, dog or spaceship capsule. (unless you are going to host The Pirate Bay this way, or automatically dump passwords on Pastebin for example)

Getting started:

Here is a shopping list to start with (the prices are an indication e.g. cheapest (or similar) on eBay/Buy now):

  • Raspberry Pi model B [$35]
  • Bag, suitcase, storage box (as long it’s ventilated) [0$]
  • 3G router with dongle[*] (I used: EDUP EP-9501N router (since it has a battery) and a Huawei E220 3G dongle) [$40]
  • 2 USB solar panels (w/ internal battery of 1000mAh, or 1 of 2000mAh. The higher, the better, the bigger, the longer battery life. [$6-$20]
  • USB Y-split cable (if you have 2 solar panels) [$1]
  • SIM card with huge internet bundle [$?, varies per country]
  • Dedicated VPN (for remote port forwarding, since you have 3G)[$10]
  • Some IT experience [priceless]

Step one:

Get all the items, charge the solar batteries and the portable router.

Step two:

Remove unnecessarily services and applications. I’m assuming you’re controlling your Pi without a display and using SSH. If not, connect to your Raspberry Pi over the network (connect it to your own network).
Now, set-up the VPN, and make it automatically connect. Write the IP address down of the VPN.
Make your server software ready, eg. Apache to use this IP.

Step three:

Configure the router. On most EDUP models, this is 192.168.1.254 or 192.168.1.1. If you’re unsure, check the sticker on your router.
Connect your 3G dongle and insert your SIM card. Next, set the APN, username, and password.

Also, there are some 3G dongles that do not need an external power source to be connected to the Pi. Use Sakis3G on your Raspberry Pi.

Step four:
sudo poweroff your Raspberry Pi.
Now, connect the 2 charged solar batteries using the Y-cable.
Connect the Ethernet cable from the portable 3G router to your Pi, and also connect the 3G device.

Step five:

SSH into the IP of your VPN. Now you are connected to your Pi over 3G!
Put it in some case or bag, and travel for 5 hours without power loss if you’re lucky. For more power, upgrade the battery inside the router(this is the first device that will die, thanks to the 3g connection), and connect the router via USB to the Pi. This shouldn’t give any problems, since you have multiple power sources.

And of course, it is not completely low budget, but you can easily get the prices down if you know what and where to buy.