We’ve been working for three Years on the virtualization technology and Compute Engine, So it’s not a copy of what you Can find in other places It’s really unique to Google So from the bottom up, You know what [ INAUDIBLE ] was Explaining that, because of our scale, we had to Build our product, We have to design our server, We have to build our Network layers And our machines are the same. So the operating system – we – what we find the best we Strip it and then we customize it because this Is our own hardware, So at all the layers this is Google-Only technology, We don’t use Apache, We don’t –, maybe for Internal stuff, But for the project we have Rebuilt web servers file system, as you can see, really From the bottom up, So that’s why this Virtualization layer is completely optimized for our Platform, our servers And what we see and I hope That you will try Compute Engine is that we managed to Have a very low footprint, Which means between the physical Performance and the virtual performance there is Always a little loss because there is a virtualization layer.
Running on the machines Here, the footprint of this loss – Is very, very, very thin because we’ve been Optimizing that So you almost have the Physical machine’s performance And what is also important, is That we’ve been working a lot that was in the early Specifications on the we call the noisy, neighbors problem. The noisy neighbors is that with Virtualization technology you have on public cloud Usually you have a virtual instance running on a Physical machine, But you are not alone – Of course, You have neighbors, so other Customers that are running their own virtual instance on The same physical machine And, let’s assume there is a bit Of overbooking, when your neighbors — so co-locating on this Physical machine — are hitting hard, So maybe they have A peak of load, You will see your own Performance dropping That’s something we want to Avoid with Compute Engine We’ve been working hard on that To do live migration of machines when a physical machine Becomes noisy so you have your performance from Day, one to the end, And you have constant Performance over time You don’t have performance.
[ risks ] The boot time and the Availability time as well is really important. You will see in the demo. You have your machine in 20. Seconds more or less so 20 seconds from the time you click New Machine to the time you can SSH the machine, So it’s not 5 or 10 minutes, It’s 20 seconds And the boot time. Is really fast, Like it’s less than 20 seconds, Because there is a staging provisioning, Time before Our objective is always To have zero but it’s impossible, But we’re trying to go.
Below one second, So we are working really really Hard on Compute Engine, There is a lot of unknowns at Google I/O that we made Like we are providing now load Balancing so level 3, We are providing advanced Routine, We are providing automation, So there are a lot of features. Coming on the project, So a few slides And then demo, What are the use cases again? About Compute Engine, So the first one, hence the Name, Compute Engine was compute.
The idea of you can Launch any number of virtual machines using Linux, So today we support Debian Centos and we have a core OS, which is GCL — Google Compute Engine Linux –, which is like Ubuntu Let’s say And we’re going to support More operating systems over time, I can’t disclose in a public Audience what we’re going to support, but in the following Months, we’re going to support more operating systems; That you all know The idea is that it’s not Only the machines, It’s really about again, as Always on a cloud platform, the network, The machine-to-machine networks, Are incredible In less than one millisecond You can transfer terabytes at a terabyte per second.
Of course you have The physical limit of the hard drive, What I’m saying is that there Are almost no limits from the network perspective, The storage –? So there are two new Storage mechanisms – We’ve been talking about cloud Storage, but you will still have the local storage On the machine That will live with the machine, Meaning that when you create the machine, you Have this disk, It’s usually hundreds Of gigabytes, It depends on how many cores You want on your machine And I will show you after Doing the demo, But you have also something I Mentioned earlier, which is a persistent disk, which is a Shared storage like a SAN, if you want across machines And there’s a lot of tuning, As always, you can use The web REST API to create your machine.
You can use the command line. So gcutil, similar to the gsutil I showed earlier Or you can from the UI Create the machine And that’s something: I will do as well, Because I need something: Quite visual for you, I have talked about All of this — scale performance value. So that’s the overview. What you Get with Compute Engine So of course you get What’s right here, so the virtual machines, But you get a private network.
Within Google And you can do sub-networks as Many as you want set your firewall rules, And I hope you all understand. That you are root on this machine, so this is your machine. Running in Google infrastructure, That’s really the idea, So you do whatever you want And they are of course connected. To the internet, so you can host an. I don’t know. Website on Tomcat, if that’s what you want to do, Maybe you can’t read: Very well So this one was persistent.
Disk This one was local disk And this one is cloud storage. So there’s three main storage. Mechanisms we provide And there’s three access types. So the command line, the web UI and from the code So the API itself, so either the REST, HTTP API or using A client implementation — again Java,.NET PHP You name it So what we call the project. Is where all the machines will be linked? It’s not physical stuff.
It’s really a way to organize. Your different applications, So in one project you have All your machines, You can go to another project, If you want to separate And within this project you Have your own persistent disk, local disk that comes with the Machine network et cetera And each project has an ID So the machine –. So, as I said, Root access. There are stock images you can use. The idea is that you could Build your own images, We are running a modem Processor, of course, so this is Intel Sandy Bridge.
These are the virtual cores that You can have from one Then, even now we have Launched at I/O new, very small modems Like if you do dev, we call Them small and micro, I will show them, But basically you have from One to eight virtual cores And per CPU: that’s The RAM you get And we have various profiles, So we have what we call the Standard, which is this ratio? We have high memory, so you Have more memory per core And we have high CPU, So you Have less memory per core, I don’t know what happened Next MALE SPEAKER, [, INAUDIBLE, ], BASTIEN LEGRAS, Not yet no! The question was: can I create A custom machine from the API specifying the number of CPUs You have to go into the types I will show on the demo.
What Types of machines we provide MALE SPEAKER, [, INAUDIBLE, ], BASTIEN LEGRAS. That’s the High memory profile, So I will show you All the profiles, I think you will see that We are trying to address the main use cases. It’s high memory profile. What You’re talking about So let me finish with this slide. So I can show you that, right after So external IPs, of course, you Can choose to add your own IPs, which are external IPs For this machine to listen on internet, We are doing, of course, one-to-one NATing and firewall.
I will so everything On the console, This is the persistent disk. So of course there are zones I forgot to mention, but when You create your machine, you choose if you want to deploy it. In US, East US, Central or West Europe, where we bring more Availability zones, of course, like in APAC Because we are — today, GC is not deployed. In all the data centers that we have So it’s only the first ones, We are bringing more zones.
So you can deploy your application closest To your customers And everything is encrypted in Time of storage at rest and even machine-to-machine Transfers are encrypted, which gives you another idea. Of the performance We are not even slowed. By the encryption, So the ephemeral disk, so the Local disk, if you want lives and dies with the instance, That’s the idea. You create your instance. If you’re in a compute scenario, You don’t care, what’s on the machine, You just create it, Then maybe you want to set up A launch script that will configure the machine And the machine will do Some computation, If you remember the use case, Earlier with mobile and back-end and VM engine From App Engine, That’s typical, [, INAUDIBLE ].
You create the machine, he’s Going to do his job and then you destroy the machine. And you don’t care what’s on the machine So for that kind of use case. You can use the local disk because it will die. With the machine, If, however, you need to persist, The data generated by the machine, then you want to Use the persistent disk And you can boot a machine From the persistent disk, That’s really key because you Want to minimize your cost! So when you don’t need the Machine anymore, you will destroy it As it takes 10 to 20 seconds.
To get the machine again, there is no need to let It slide the light When you go out of the room. You switch off your light. That’s really the idea. Of open engine, You don’t want to spend money. For a machine that is doing nothing of course, So that’s why you need to manage Your data, of course, And the cloud storage we just Talked about cloud storage, That’s of course integrated. With Compute Engine Again easy to store terabytes Of data or petabytes of data from Compute Engine – And that was the case in the big Migration use case I was showing before And the three access doors that You have — so UI command line and from the API.
I won’t go into too Much detail here: We’ve covered that More or less So Compute Engine is In most of our big architecture diagrams, So you see it here: it’s a Hadoop MapReduce use case Here it was the thing Of this morning I was presenting when you want To do batch processing for your mobile app on demand. Et cetera, et cetera, This is an analytic pipeline. So you have a lot of data coming in You use App Engine for Mapreduce and then you want maybe to also use Hadoop To analyze this data Compute Engine is everywhere, I mean everyone needs virtual Machines at one point, It’s very, very rare that you Don’t need it in your project And before going into the demo, So we already have, of course, an ecosystem Right Scale is one of The first partners, So they know very well Compute Engine They have their own image.
Because that’s the idea, You can start from a cannot Machine so a stock image, then you can customize it And then you can take a snapshot. Of course You put it on cloud, storage and Then you will reuse it to reboot a new machine, So you have your own machines. In your project So Right Scale, they are Providing their own image And they added onto the image A connector that connects to their central console The idea is to have elasticity So scalability up and down at the infrastructure, Level We provide that with App Engine, As you know, it scales Up and down You don’t have to worry about How many machines you need to run in your app When you are at the Infrastructure level you have to manage that So do I need to put More machines behind my load, balancer.
So do I need to remove some Machines, because now I don’t have the demand for All these machines, So Right Scale, is doing That for you, You just configure You say I don’t know you Define some threshold, You say at above 70 % of usage, I want you to add five more machines And the rest of the inverse Rule so when you go below 50 %, you remove three Machines, You define your own rules and Right Scale will manage the scalability so from The console directly on Compute Engine – And they do also a lot of Monitoring and stuff, These are other partners.
Technical partners, MapR, is a commercial Hadoop. Implementation, I have a project with them. I was mentioning [, INAUDIBLE, ], Earlier They are using MapR for a Hadoop Project to analyze the traffic on the website, And improve their conversion ratio, So it works very well. We broke the [ teraflop ] record. It was one minute something With Compute Engine It’s now less It’s like 56 seconds. So it’s to give you an idea.
Of the pure performance of Compute Engine, I won’t go into that. It’s the big data landscape. Let’s go on the demo for The last 15 minutes, So I have to sit down again Sorry And then I will take of Course questions, So let me clean that up, So you start from here. Why is that there? I hope you see. So this is the console home Page for a project, The console: let’s go, So you can understand really how it works.
You go on here With your Gmail account. You Activate the services you want to use and then you create Your project – And here I am logged with My co-op account Google, So we see we have some projects. Internally, that are shared with me, And I have access to this one Which is our demo creator Here? You see the services That are activated for this project. The project is like a team. You can share the project with Other people in your team, like you, are sharing a doc in Google Docs and give specific access to everyone So App Engine, so this means That on this project, I guess people are using some App Engine instances Compute Engine and we Will go into that Cloud? Storage, Big Query! Some APIs [, INAUDIBLE ], There you see which services Are being used So, let’s focus on Compute Engine and then let’s do a quick tour of the Administration.
Console Here you can see a basic Monitoring graph, That’s just showing nothing. Is happening, It’s a demo environment. There is no external user And we will give it some loads. During the next demo, You can see the network traffic As well that we are monitoring and disk traffic, You will see here –. Let me zoom –, that’s a sample machine. So that’s the name! Of the machine, The machine is running, You see the profile of the Machine — one CPU 3.
8 gigabytes of memory. Where is the machine? So it’s in US Central This one, How many persistent disks are Attached to this machine, What is its external Ip address And on which network? So that’s a default network. But you can create other networks with other rules on Which the machine is deployed And the images that the Machine is using Here, you see, there are Some Ubuntu GCEL, So that’s a few machines.
We’ve Got on the project Machine doing fractal Stuff, which is the point of the demo after You see there are some Write images, so I guess it’s Right Scale And you see all your machines And which state they are et cetera. So now, let’s focus On a machine so that one for example, You can put some tags on your Machine, if you want to organize your machine, That way, Again, as I said, you see the Zone external IP internal, IP disks that are attached, To the machine, So this one is mode One 10 gigabytes, It has read and write access, You can add metadata, which is Useful going to want to use a launch script et cetera, With the API And then you will see the Permissions, the machine has with all the services, And you can [ recognize ] storage test use, Big Query.
So that’s why I said we have Integrated the projects to each other, So let’s create one machine. Right now, so you can see how it works: Test Bastien, One and Description tags metadata, so you add whatever You want here There you define where you Want the machine to be So, you see I can deploy in Europe or in US Central Today, more zones Are coming in So let’s go in Europe West B, And back to your question the types of the machines, so you Can see you have the new machines which are micro and Small, so very small, if you don’t need power, I don’t know.
It’s like a machine that is doing some networking stuff. You have the high CPU profile, Where you see the number of CPUs and the low memory Best [ view ], But if you need a lot of memory, Here you can see high mem profile, –, two CPUs, 13 gigs or even Eight CPUs 52 gigs, That’s the types of machines. You cannot yet define your own Type like I want a three CPU or four CPU and 50 gigabytes. You have to choose the Closest today MALE SPEAKER So with the Restful API: is it only possible to specify If [ INAUDIBLE, ], BASTIEN, LEGRAS Yeah exactly And from the API, you can list Which are the templates? Because we are adding New templates, So you can release the templates Dynamically and say: OK now I will take that One because it’s closest to my requirements, So that’s a dynamic List actually – And you have other machines with What we call a scratch disk or ephemeral disk That was in my slide that local storage – I can show them by the way.
So when there is the –, so you See the machine and it’s local disk. So let’s take I don’t Know anything — high memory profile. Then you can boot from an Existing persistent disk, As I said earlier, if you need To work on data, you have persisted earlier or you can Just take the scratch disk or a new disk that comes With the image, So let’s do that, Then you have the list of the Stock images we’ve got So we build them frequently.
So You can see here we have –. This is the US, so it’s 7th of May 9th of May for Debian, CentOS and GCEL here Or you can also save –. Sorry, I hope you can see If you can’t see Raise your hand Or you can save your Own images So other guys in my team. They Chose to create a Tomcat 6 type of server, So it will have Tomcat already Pre-Installed, Because when we did the Snapshot we had already installed and configured Tomcat So let’s take a standard Machine And then you can add Other disks, If the machine has to work, we Shared data – you can add other disks that you have Already created, We got in a network, So you will define in which Network you want the machine to belong, So let’s keep the default And then external IP.
If the Machine has to have an external IP And here the access. So what it means is that if you Want the machine to have access to storage, read and Write or no access or to test use, we will pre-prog This kind of access, So in your bigger projects, if You’re using App Engine in test use – and you want this Machine to work with test use, then you will give it Access right now, Let’s create the machine.
So that’s where everyone Cross, your fingers, But basically so It’s loading It’s going to the stage the Machine into Europe West, which is 20 milliseconds, From here I think, And then we will see it Directly in the list, I have an error, but it’s Not the same page, It’s not my concern. Don’t worry. It’s still creating The machine It’s another — and that’s the demo. I will do By the way, right after hoping, this one will work In this demo, we’re going to Create one – and here we’re going to create 16, so 17 Machines altogether using –, this is an App Engine app And it’s going to use the Rest API, of course, to create the machines, While it’s still creating There, the machine, I think in a few seconds, the Machine should be there Here, it is It’s being created, So I guess it’s alphabetical Here.
It is coming in In Europe West B, The machine is there OK, quite fast. Finally, I didn’t measure, I think it was 20 seconds. Something like that And the machine is available And if you want to connect to The machine right away, you have the REST snipped or you Have the SSH command line you can directly use in Your terminal, So gcutil is the equivalent Of gsutil, but for Compute Engine, Let’s clear that I just passed the command line: Gcutil last version: That’s the project; — Google Platform, Demo SSH In that zone, the machine I just created Normally, what would Happen it would ask for a login password, But no because we have Integrated everything, So this is using OAuth.
It’s not visible. I’r going to zoom like This, maybe MALE SPEAKER No, but You can just — BASTIEN LEGRAS, Ah sorry MALE SPEAKER, So you can Enlarge your fonts Yeah BASTIEN LEGRAS! That’s! Ok! Now, Sorry MALE SPEAKER, [, INAUDIBLE, ], BASTIEN, LEGRAS, Good idea, Good idea, [ Fredo ], So I copied the GS2 Command to SSH And, as I said, I didn’t have to Enter login password, which is ultra-painful by the way Because what I’ve done is that the first time you run gcutil It will send you to a URL.
You go to that. Url In your browser, This URL will ask you to Authenticate against Google And if you authenticate Successfully, it will give you the OAuth key that you copy Paste back in gcutil, Which means that I have Authenticated gcutil to cloud platform on behalf of me So now this gcutil installation On my computer has access to the machines To which I am authorized in the project, You understand that It’s very convenient because no Need to log in to remember password or whatever, And it’s even more safe.
So we are on the machine. So usually the demo stops there. Of course, because I don’t have much things – To show here I don’t know, Let’s check if the memory Is the right one? So we have our 26 gigabytes. What about the CPU? I need to –. I have a hard time: — OK. CPU info. So here you can See it’s really big. My CPU numbers zero. One two three: So there are four CPUs As we asked And you see Intel Xeon CPU, You see the technical Detail of the CPU, So the machine is there I am root.
I can do whatever I want. On the machine, So let me do another demo before We finish and take some questions. The demo is nice and visual I hope it’s going to work. Ok, I don’t want to mess up Sorry. So this is an App Engine And this is a demo that has been I think showcased at I/O. This is App Engine And the idea of this demo — This is a very technological demo. The idea is to start Vms using the API, That’s what I’m going! To do right now And I need to show In parallel, Sorry, It’s for the demo to Be nice to see Sorry, I will have to do Something like this, So I want you to See that list And then the idea is To start the VMs, As you can see, it’s Coming blue here So on the right we have a Cluster and on the left, a single instance And here on my poor console Because I have [ INAUDIBLE ], we should see as soon as it’s Going to be refreshed, Sorry, I’m going to [ force ] refresh.
Normally it’s refreshed. Ok, we’re going to see The fractal cluster –, which is the one I’r creating — that is being started. Ok, this is this one And on the right, the machine Once they are green, the idea is that the machines Are ready to SSH The program — so App Engine Still App Engine is going onto the machines provision, a Little program that will run some fractal imagery, So it’s pure scientific computer, Calculation, so it’s using a lot of CPU And then we will display — if it works, because I am waiting also for the single instance — and display some fractal Imagery And one is a distributed: Calculation, The other is single, So we see how fast the machines Work together because that’s the type of use Case that requires a lot of network also, So the machines are there.
That just got created, I hope I can show So I don’t know why dimension On the left, the single instance failed, But the idea is that the machine On the right, as you can see here, zooming out have Been able to create their own fractal And then it will mark on the Right the render time And it will calculate As long as –, when I zoom here with my two Fingers it’s calculating the next layer of the fractal.
So sorry half of the demo Here is not working The idea –. Imagine that on the left you Have exactly the same thing, but only one machine is Trying to do the job Here we have eight machines, Doing the same job, So what we see on the left? Usually, is that the average render time is much bigger. Because it’s only one core And you can zoom You can zoom, it will calculate calculate So as soon as I zoom.
It’s Recalculated the fractal and rendered that in a distributed Calculation, Because all the machines are Doing the calculations – And it’s really pretty – You can see the [ colors ]. So that’s the idea. On how fast, So how long did it take? I don’t know 30 seconds And I could say three eight. I Don’t know 128 machines, it would be the same Because it’s done in parallel. We do that every Day at Google We have millions, We don’t say exactly how much But are in the millions range in terms of size of cores And then you just stop and the Machine will be destroyed on the left, So App Engine is asking –.
Ok, they are dying right now. It’s asking for the –, I don’t know where they are, They should go. Maybe they already disappeared. Because they are already red, They already disappeared. But usually you see It live also Here you see the — OK here they are. You see It’s spinning right there. They are dying right there. One after one And now they have disappeared. I hope you caught it, But it has destroyed the Machine and then it’s ready back for another demo.
Ok, I’m done for the [ pres — ]. I can take some questions if You have before we go into the coffee break. Yes, We’ll try to pass the mic [. We have ] one mic Thanks MALE SPEAKER, Ciao Bastien, It’s more a pricing question. If I have some data some files, Terabytes on the Cloud Storage and I launch 10 virtual Machines which share the data on the cloud storage, Do I pay for this internal exchange, BASTIEN LEGRAS, So Good question: So we don’t charge for cloud Platform service to cloud platform service network Exchange, So what it means is that you can Do any traffic between App Engine and Cloud Storage and Compute Engine between Compute Engine machines between Compute Engine zones inter-network between Compute Engine and of course, Cloud Storage, It won’t cost you anything.
In terms of network, What you will only pay Is your 10 terabytes MALE SPEAKER, The storage BASTIEN LEGRAS, So one terabyte Is $ 50, so you’re going to pay? Maybe $ 500, I think I can show the pricing [ slide ] And then you will pay For your machines, Maybe you want to have a look Because the pricing question, I think, is interesting: For all of you, It’s fully public. We have nothing to hide. We are transparent, So cloud.
Google.Com, as you Understand is the site to go. This is everything From there. You can Find everything — the project, the project Explanation: data sheet the website to developers where You can see all the API documentation And, of course, there Is the pricing and pricing by project So App Engine pricing? Is the Most complex because we want to be really fair to really Price only what you use As you can see, you will Be priced by what you are using exactly If you are using front-end Instances dynamic et cetera: you have the price per hour, Datastore API, we price Every 100k operations We have a very [ fine, ] rate For that kind of pricing For storage, it’s very Easier to understand There is always a free tier So you can try for free, You have a free tier Five gigabytes of storage et cetera, So you can try and Use storage: We have two types of storage, So the most replicated and what We call durable reduced availability, less replicated, Which is, of course cheaper? We do pricing in Euros if you Want to pay in Euros with a contract, et cetera, And we price, of course, Depending on how much you need, So, if you are in the 100 Terabytes range we will price by gigabyte.
This is a gigabyte per month. Price that one! So if you had 10 terabytes You can see it’s 54 — so $ 540 per month And for the network, as we say, Everything that goes into the network is free. Of course, we charge for the Network so egress network from Google to the internet, To your users And that’s the type of price, But there is no Service-To-Service cost So internal Google transfers Are free and then some operation costs, But every 1,000 application Operations is $ 0.
01, So basically you won’t see it And Compute Engine very key. So that’s really important, That’s something we announced At I/O, So if you use the machine six minutes, we’re going To charge 10 minutes, If you use the machine 10 Minutes we charge 10 minutes If you use it 11 minutes. We charge 11 minutes, You see, so there is a minimum Set of 10 minutes and then we charge by the minute And then for all the machine.
Types you have seen earlier when I was creating a machine You have a price depending if the machine is in Europe. Or the US And you have a price which is Per hour, so you can divide by 60 to have the by Minute price And you have a price for each Type of machine, of course, So whether you need or not a Local disk on the machine, so that’s the size of The local disk, The high memory profile so You see two cores 13 gigabytes.
Instead, Of two cores, So it’s twice of it almost Up to 52 gigabytes, So here is everything [, INAUDIBLE ], And here it’s a ratio, CPU or Memory, whether that is bigger, so less memory, Per CPU, If it’s an [, HPC ] use case, you may want To use that – And you have the very new ones – Which are the small ones, the micro and the small, So we are at $ 0.021, It’s very small about $ 0.2 an hour And again network pricing, Between our services or in the same zone is free.
We charge only external Network And then we reduce depending On the amount of network, I’m not the commercial but as Kristoff said earlier, if you’re interested in a Professional contract with support you can see, there are Several support packages, which is there So we do up to Platinum with Dedicated account manager for you et cetera, But we have the free one. So you Can, of course, access to the [ INAUDIBLE ] to the Stack Overflow et cetera, There’s silver, so you Can do ticket You can call call in, And Gold — and Gold is like.
You have a Committed response time And you have all the various Commitments here at the pricing detail And if you are interested in That type of support, if you are in production, you have Customers, you need to make sure it’s going to be up. It’s Not just test and development, then you just contact Us right there That will arrive on Kristoff’s table So very simple: Everything is there And on the articles, everything Is if you search for Google I/O sessions, you will Find them right here And you can choose your track.
And, of course, my track is Cloud Platform. Ok, I think we’re good. Maybe you want to have your Coffee break now: [ APPLAUSE, ], BASTIEN LEGRAS. Thank You very much
Jimmy's Internet Marketing Business Blog 