i think the problem is your trying to over think things peter and are making it seem more complicated than they are/should be.
from a windows pc.
all you need to do is download the ubuntu live cd iso.
download and use linux live usb creator
start the startup disc creator needed to put the iso file on the 4gb usb stick, select the 4gb usb stick as disk to use
slide the slider where it say the documents and settings will be saved in the reserved extra space all the way to the right using up the full 4Gb usb sticks space.
make the startup disc.
or download the unetbootin for windows and select persistence.
far as i know peter that's all you need to do to have a live usb stick with persistent file storage on a 4gb usb stick.
now just plug it in boot the laptop/pc use the F12 option to boot to the ubuntu live usb stick and away you go.
https://help.ubuntu.com/community/Installation/FromUSBStick
https://help.ubuntu.com/community/LiveCD
THERE IS NOTHING ELSE YOU NEED TO WORRY ABOUT SECURITY WISE so don't go trying to over think or complicate things now you have a running live usb stick with persistent file storage.
ALSO note the usb stick is not meant to be used as your normal desktop, yes use it to do "sensitive pcing" but after that's done then just reboot and use your normal operating system.
if you really want to run a secure linux operating system which you can keep updated then using a usb flash memory stick is NOT the way to go, they wear out much to quickly with all the writes and are also way slow.
for that easiest way is to buy a dedicated usb3 hard drive like this :- http://www.amazon.com/WD-Passport-Essential-WDBACY5000ABK-WDBACY5000ABK-NESN/dp/B008C4KNOO/ref=sr_sp-btf_title_1_15?s=electronics&ie=UTF8&qid=1396963502&sr=1-15&keywords=usb+hard+drive
use the ubuntu live usb stick you've created to do a full install to the new external hard drive, making sure it's bootloader is put on it too and not on the hard drive in the laptop/pc.
ok not going to be as fast as if installed onto the internal hard drive if you don't have usb3 ports but it will also not wear out like the usb flash memory stick does.
this then should be able to be used anytime you want to run the linux operating system in exactly the same way you've been using the live usb memory stick on both your laptop and pc.
i caveat to that tho is if you install any proprietary software like graphics drivers then it may not work as you expect on the/any other device. for example if the pc had nvidia graphics card and you install the nvidia driver then when you try and use it on the laptop the laptop may have intel/ati graphics so the nvidia drivers won't work. now it may not boot to a desktop like you expect and leave you at a command line, try booting back on the desktop again tho and it would work as expected.
it may just boot regardless too tho, i know i've swapped and booted suse linux hard drives around without any problems even when i had installed proprietary drives for one machine that the other machine didn't have and it still worked just fine.

which I think is a way to get one to pay
one last thing peter, you've mentioned paying for linux os in a number of your topics now, so please understand linux NEVER needs paying for.
if you are told you need to pay to use a linux os then you are being ripped off plain and simple.