47

I want to prevent potential theft of my web application (source code + database) by my local hosting company, that I don't fully trust for some reason (but have no other choice but to use as they give, by far, the best latency over here).

I'm planning to encrypt a partition with cryptsetup, and store the web directory + database files there.

The only issue is that I'll have to unlock the partition on every reboot (even uncontrolled ones), before I can start the web & database servers; but I guess I'm ready to live with that for now.

However, as far as I can read all over the web, encryption keys are stored as is in memory, and can be read given physical access to the machine, even from a dedicated machine using cold boot attacks. On a VM, I guess this is even simpler, as the hypervisor can take a snapshot of the whole server state (storage + RAM).

So as far as I can tell, it should be pretty easy to take a memory dump, and locate the encryption key at any time.

Is there any way to prevent this?

I know that controlling the hypervisor gives a huge advantage to someone attempting to steal data, and as such I'm not looking for a perfect solution here; but I'm rather trying to make it as hard as possible to get access to the data, so that it would not be economically viable to spend time & resources to get them.

I'm not really optimistic here, as encryption keys must be stored in some form in memory for decrypting, but I hope I missed something.

Edit - clarification

From my comment below:

I'm sure the hosting company's hypervisors are not modified in any way to perform any malicious operation; their business is hosting, not stealing things, and the company is relatively trustworthy. What I'm trying to protected against is someone bribing an employee to sell them a snapshot of the data. As long as the encryption key cannot be recovered from any snapshot the stock hypervisor can do, I consider the solution good enough for my use case.

Improve this question
8
  • If homomorphic encryption ever becomes viable I'm sure some hosting company will offer secure VM hosting.
    – user
    Commented Aug 26, 2019 at 17:27
  • 5
    @user Homomorphic encryption isn't magic "compute things on untrusted media" sauce.
    – forest
    Commented Aug 27, 2019 at 6:47
  • 2
    ... why bother with the encryption keys, when your code/database is going to be decrypted when running? That is, just snapshot the decrypted code directly.
    – Clockwork-Muse
    Commented Aug 27, 2019 at 20:23
  • @Clockwork-Muse If you're talking about the executable code (opcodes) residing in memory, database server memory pages, and disk cache, then yes, at least a part of the source code + database will be in memory when snapshotting. However, I consider the source code and the database valuable only as a whole, not as a 1,000,000 pieces puzzle that one would spend months trying to make sense about. As long the work to recover anything useful out of the snapshot is non trivial, I'm happy.
    – BenMorel
    Commented Aug 27, 2019 at 21:07
  • 2
    Have you tried asking the company what measures they take to prevent a malicious individual employee from doing something like this?
    – smithkm
    Commented Aug 28, 2019 at 17:52

9 Answers 9

Reset to default
110

You can't, plain and simple. If you don't trust the hosting company, you don't host with them. This is law #3 from 10 immutable law of security:

Law #3: If a bad guy has unrestricted physical access to your computer, it's not your computer anymore.

The hypervisor always have privileged position over your virtualised machine, you can't protect yourself from rogue hypervisor.

One way you can do to benefit from their network location but still keeping your code secure is to do a colocation hosting. In a colocation hosting, you bring your own machine that you put in someone else's datacenter.

Basic colocation itself should discourage most opportunistic attackers, but a targeted/determined attacker might connect into your machine anyway. If this is a concern for you, you should put your machine in a tamper-evident/tamper-proof server casing, so that the only unlocked port to the outside world is the power cable and network cable, and you encrypt all network traffic going into/out of the machine. This way you can put your machine in their network and benefit from the network's low latency, and they won't be able to steal data from your server unless they break the server's chassis.

If protecting your data is very important, you would also want to do a regular audit that the case hasn't been tampered with. If you have infinite budget, it may also be possible to design tamper proof chassis that have various sensors that can raise an alarm and initiate shutdown (discarding all sensitive/encryption keys) if they detect that the chassis had been tampered with, but of course these gets expensive really quickly.

Improve this answer
13
  • 36
    ...and the next step up, of course, is to set up your own hosting facility, complete with whatever physical (locks, doors, ...), electronic (alarm systems, surveillance cameras, ...) and human (guards, receptionists, ...) security elements are needed to ensure a proper level of security. Of course, that gets expensive real quick, but hey, what's the price for a good night's sleep anyway?
    – user
    Commented Aug 26, 2019 at 9:37
  • 10
    @Benjamin: TRESOR assumes that the hypervisor is trustworthy. A rogue hypervisor can fake it so that your program thinks it's running code on the CPU, but turns out your instructions are actually running on an emulated CPU whose behaviour and debug registers is fully controlled by the attacker.
    – Lie Ryan
    Commented Aug 26, 2019 at 10:10
  • 9
    @Benjamin: the host machine is privileged over the guest machine. It's pretty simple for the hypervisor running on the host machine to read the debug registers of the guest vcpu. This isn't even difficult as many hypervisor supports guest debugging API that includes reading debug registers among other things. Even in real hardware, the attacker can attach a JTAG debugger to read out the debug registers. These type of attacks sit well outside TRESOR's threat model, but will be relevant to yours.
    – Lie Ryan
    Commented Aug 26, 2019 at 12:48
  • 17
    That. If I were that rogue hosting company employee... well, I wouldn't take the bribe, but if bad guys had a member of my family hostage, I absolutely could obtain a register dump. And could do it in a completely undetectable way, assuming that the hypervisor in question supports live migration or (full-system/live) snapshots. Dump the (emulated) hardware state, copy that data to personally-owned media, and come back to analyze it later. And I'm not a pentester or RE specialist, just a systems engineer with some sysadmin and C experience.
    – Charles Duffy
    Commented Aug 26, 2019 at 17:32
  • 1
    Strictly speaking, you could use remote attestation and run all the sensitive computations in an SGX enclave (and SGX disabled JTAG), but it would be difficult to accomplish, needing a complete rewrite of the hypervisor. At least it would move the trust from your VM host to Intel's hardware. The only full solution that I'm aware of is vCage, which is commercial and quite expensive. The only issue with it is that it does not disable JTAG, as far as I am aware.
    – forest
    Commented Aug 27, 2019 at 6:40
16

so that it would not be economically viable to spend time & resources to get them.

I hate to break it to you, but you simply aren't that important. No-one knows you or your web app. So it's already not economically viable.

Consider the cost-benefit. For the hosting company, if this happens once, they'd hemorrhage customers. So they're going to have proper logs. Any employee caught doing this will not only be fired, but also arrested. So the bribe would need to be serious. We're talking millions, because it's got to compensate the employee for being imprisoned and never being able to work again when released. And if that's the bribe, your company must be worth at least 10x that. That values your company in the tens of millions, and more likely hundreds of millions.

Honestly, is your company worth that? Everything you've said so far says it isn't. And if it is, you have no reason not to run your own servers.

Improve this answer
8
  • 22
    -1 for I hate to break it to you, but you simply aren't that important. The people who assume that are the people who end up being screwed over the most. Even if OP is not that important, OP might have access to something that is (maybe he used to work at a major ISP, and an attacker is looking for ex-employees who might have old credentials saved).
    – forest
    Commented Aug 27, 2019 at 6:43
  • 2
    You're perfectly right, the company is rather small and the project is not that important. However, the database contains a decent number of customers from the whole island, and the app took many thousand man-hours to write. Plus, I wouldn't trust that the company is actually keeping logs of their employees' actions or that the employee in question is risking anything at all (let alone prison) if he did break into our server. Therefore I wouldn't be surprised if someone actually sold a snapshot for a couple thousands, not millions. THIS is what I'm trying to protect against.
    – BenMorel
    Commented Aug 27, 2019 at 9:58
  • 3
    @Benjamin Then you need to check that the company is keeping those logs, and you need to verify that the audit mechanism exists for you to see all administrative actions on your VMs. Especially if you're in a country with privacy laws (GDPR is the obvious one; do you do any business with European clients?), your clients' security depends on your hosting company's security. And if you really have that low an opinion of your hosting company then you shouldn't be using them, regardless of their latency.
    – Graham
    Commented Aug 27, 2019 at 11:50
  • 4
    @graham the assumption that you won’t be attacked if you’re unimportant is unsafe
    – Gaius
    Commented Aug 27, 2019 at 12:04
  • 3
    @Gaius The watering hole attack isn't relevant to his situation. And whilst I agree that it is unsafe to assume you won't be attacked, safety is always a function of cost and probability. As an engineer where "safety" on my projects has meant "people die", I'm more aware of that than most! :) The OP needs to decide what is "safe enough" given the probability of this specific attack vector, and unless there are other factors, the probability looks vanishingly low. Of course if his company makes it big then he needs to reassess, but then he can probably afford better hosting too.
    – Graham
    Commented Aug 27, 2019 at 13:04
13

In theory, you should be able to use the trusted hardware features of modern CPUs to run your disk encryption, or even your entire VM, inside a tamper-resistant part of the CPU, having all the data on disk and in memory encrypted with keys that are only accessible inside that tamper-resistant trusted enclave.

While exposing Intel's SGX trusted computing system to VMs doesn't appear to be possible on stock hypervisors, AMD has a feature called Secure Encrypted Virtualization, or SEV, which sounds like exactly what you are looking for: you can set up your VM in such a way that it is protected from the hypervisor with keys known only to AMD or someone who is willing and able to disassemble the host CPU.

Unfortunately, it's unlikely that the particular hosting company that you need to use provides AMD SEV-supporting hosts or makes the feature available to their clients.

Improve this answer
5
  • Interesting, thanks. Indeed there is no chance that this hosting company uses this, though.
    – BenMorel
    Commented Aug 26, 2019 at 22:01
  • 3
    Anyway, regarding the SEV thing, quickly reading about it, it doesnt seem its provable that the host is really using it. What would stop a malicious server running the VM on an emulated system that pretends to use SEV but just fakes the appropriate instructions?
    – Vality
    Commented Aug 26, 2019 at 22:12
  • @Vality if you hosting provider gives you access to hosting machine and you are the one who setups server with SEV, this might work.
    – stiv
    Commented Aug 26, 2019 at 22:27
  • @Vality As described in github.com/AMDESE/sev-tool#guest-owner it looks like you encrypt (the sensitive part of) your VM with a key shared between you and the trusted hardware in the CPU, so the data can only be used in a guest running the image you specified. Image verification is accomplished somewhat like secure boot, I think, except instead of the TPM verifying measurements about the boot process of the host, you yourself verify and approve measurmenets about the boot process of the VM.
    – interfect
    Commented Aug 26, 2019 at 22:56
  • 1
    @Vality You can verify it just like you can verify that a TPM is real and not emulated (for a TPM, that's through its Endorsement Key). I'm not sure exactly how SEV does that, or even if AMD supports remote attestation beyond simply checking firmware, but it might.
    – forest
    Commented Aug 27, 2019 at 6:48
4

Microsoft has a possible solution for this called Shielded VMs, these are specifically intended for the attack vector that some malicious actor has administrative access to the hypervisor. An example of this would be machines deployed either in the cloud or in a colo.

The downside is that you will need to maintain a Guardian Server under your physical control. That may be overkill for your needs.

Improve this answer
2
  • 1
    Even more overkill is IBM Secure Service Container. It runs on a LinuxONE machine (cost starts around 100k, If I recall correctly)...
    – ThoriumBR
    Commented Aug 26, 2019 at 19:05
  • Unfortunately I have absolutely no control over the choice of hypervisor!
    – BenMorel
    Commented Aug 26, 2019 at 22:00
4

I don't fully trust for some reason

Trust them in what way? Do you believe they may be incompetent or do you think they may be malicious?

If merely incompetent, you'll probably get reasonable protection from the encryption you've described. It is of course possible that their host machine gets compromised and the attacker finds a way to grab the current encryption keys from RAM, but in typical web host compromises the attackers won't go that far. They'll just find an easy place to dump their malware or easy to grab passwords.

But if they are actively malicious, you cannot protect the data on the VM. At best, you could configure it as a proxy server, which could give you the latency benefits without exposing your database and source code.

Improve this answer
1
  • As clarified in the question, I was thinking of a malicious employee who would be bribed to get a snapshot and hand it to someone for money. Regarding the proxy server, I thought about this idea as well; this may be a reasonable fallback if everything else fails. Obviously not everything can be cached by the proxy, especially in a web application, but if it gives a 80% or 90% of the perceived performance, this may be good enough. This would require an in-depth review of the application, though, as it's not been built with caching in mind so far.
    – BenMorel
    Commented Aug 26, 2019 at 21:59
4

Lie Ryan's answer is the right one, but beyond colocation, which still has physical attack surface at the host's premises, one additional possibility is using the hosting provider just to get you a public-facing IP address, and tunneling that to an on-own-premises server via a decent home or business class broadband link.

If you want to get really fancy, you can even do a mixed approach where the private keys for certificates never leave your own on-premises server and the server at the hosting site only has access to generate session keys. Cloudflare offers something like this; I'm not sure exactly how it works though. I think it's even possible (not with theirs, but if you roll your own, making fancy use of cryptographic primitives) to have the data at the hosting site, but encrypted with symmetric keys that are never on the hosting site, and handing off the connection to the hosting site in such a way that only the client, but not the hosting site, can decrypt the data.

Improve this answer
1
1

The cloud ultimately is someone else's computer, and someone else's rules. If you are looking for physical access security, the best path may be to host your own physical server, or get a dedicated physical server. With all the tools available to orchestrate this today, and the likely fact that you will be wanting to host projects for a long time, it's not a bad idea..

Improve this answer
0

Your question seems to switch pace a few times, but if your main concern is someone cloning your work, the following should be simple protection:

Make sure there is nothing interesting to steal

Rather than letting the source code for everything live on their machines, work on the code in a different environment and deploy only the binaries.

This should prevent anyone from copying your work as they cannot update the binaries, if you are really paranoid you could even put a time bomb in the logic to make sure things stop working even if they copy the current version.

Keep in mind that even if a sizable amount of your code does need to live in the server, it already becomes financially unattractive if you are able to block anyone from using one or two core parts. Of course they can be reverse engineered but this is likely prohibitively expensive.

This solution of course does not offer protection against data leaks, but based on your question it seems like preventing competition is more of a concern than data becoming public.

Improve this answer
2
  • 1
    The code is written in PHP, so source code == production code. There is currently no way to ship only the compiled opcodes (that are machine-dependent), and even if there was, decompiling opcodes would probably get you something very close to the source code in this language. PHP code obfuscators are usually nowhere near production quality, and ruin any chance of debugging any issue on the server itself. And finally, I don't want the database stolen either!
    – BenMorel
    Commented Aug 27, 2019 at 15:06
  • In that case I understand it will not work for you, but will leave this answer here as it may help future visitors with the same question in a different setting.
    – Dennis Jaheruddin
    Commented Aug 28, 2019 at 9:06
0

With AMD's SEV it is possible, but your cloud provider will have to support this: https://security.stackexchange.com/a/241566/84564

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .