Author: Jimmy Dagger

Okay, brilliant thanks for making me feel old. Come all of this way, and now I feel, like some kind of I don’t know, decrepid relic good afternoon good afternoon, thanks very much for coming. It’s absolutely fantastic to be here. Thanks to Chris and everybody for the invitation, I’ve been to Singapore dozens of times, but this is the only time I’ve ever left the airport.

It’s some Singapore’s a lot bigger than I’d experienced before so where last year I actually moved from the UK down to Sydney. I took over it as head of designer company called and Sarada and we’re actually making some really exciting designs that are a kind of combination of kind of editorial and product work, we’re actually building our design and development team. So if you are interested in Sydney then and send me an email or send me a tweet, and I also consulted my own company – do designs for clients, stuff nonsenses.

Our new design this week meet Errol. My new guerrilla and our goal for the past 20 years has always been to make distinctive creative work for the web work that kind of goes beyond just making a digital product or a website easy to use, because to me, design is much more than just problem-solving. You know it’s about communication, it’s about inspiring brand loyalty and encouraging affinity, and that’s why you’ll meet Errol on the website.

You don’t see pictures of me and a good deal of my work over the years has involved designing systems and pattern. Libraries and style guides worked for clients like Greenpeace and and WWF and style guides. These tools have become incredibly fashionable. You know: we’ve got articles talks, even entire conferences, like clarity in San Francisco that have been devoted to them and perhaps the most well-known design methodology today is Brad Frost’s atomic design.

I bet there’s very few people that have not heard of atomic web design. A Brad wrote a book about atomic web design. He was kind enough to mention me in the knowledge Minh said so Andy Clark, who’s been talking about design systems and atoms before it was the hip thing to do. Thank you for all your writing and thinking, but you still not getting my dog, which, which is a real shame. Actually, because my my Instagram feed is full of pictures of Brad’s, dog, Ziggy and other people’s babies.

I’m not sure that’s normal and that’s because in 2012. Oh, my god, that’s such a long time ago, before atomic design, I introduced this idea of designing atoms and elements, and it’s this process that separates what I call the atmosphere of a design from the layout and if you fast forward to today, we’ll find that style Guide has become this umbrella term for several different types of design, documentation.

They might be static, PDF style or visual identity guidelines. They illustrate how a brand should be presented and how those assets should get used across different media. This is a an example from Audi which includes how to use their fonts and colors and their logo come back to LD a little bit later. Cuz, I’m hoping that if I mention them twice, they’ll send me a free one. You just passed my water, please J’s.

So voice and tone guidelines they describe how a a brand’s personality should get conveyed through the way that it speaks to customers. A MailChimp’s voice and tone is a fabulous really well-known example of how to imbue a brand’s personality through the language that they use front. End code guidelines for developers, they stipulate coding standards to encourage better collaboration across teams.

Sure we’ve all seen examples like this. This is IBM’s carbon design system and then finally, component or patent libraries, which commonly contain examples of how to style the atoms and molecules and organisms and templates that Brad for us talked about in atomic web design. This is from the Guardians pasted up and it’s a typical example of a pattern: library, that’s been developed for developers by developers and in my work and I suspect in most of yours.

When we talk about a style guide, we mean a combination of visual identity guides. In a component or pattern library – and these all offer something slightly different, but more often than not, they all have something in common. Can you guess what that is? They all look ugly enough to have been designed by someone who really enjoys configuring, a Reuter or router, depending on which part of the world you’re in okay, that that was a bit mean because an unattractive style guide is not going to be a problem for everyone.

I mean people say as long as it contains the information that people need how it matters shouldn’t, look or should it. You know this means that is the thing right. Beauty complements functionality doesn’t detract from it. Creative design enhances someone’s engagement with the style guide and it should amplify the content, and that means to be effective. A style guide should present that content in appealing and engaging ways and recently interest in design systems has meant that we’ve had reproductions of corporate design manuals.

These are become really popular, this one’s from British Rail, I ordered mine online and it came late. The NASA graphic standards manual describes the design of everything from a business card to the branding on a booster rocket. You got the New York City, Transit Authority, graphic standards manual. It includes the most comprehensive guide to how to use Helvetica that I’ve ever seen not making this up.

Finally, this is a mouthful. The official symbol of the American Revolution – Bicentennial graphic standards manual, explains that only how this symbol should be used, but also how it was created. This is important. I’m going to come back to this later and now there are collections of design guidelines online, including this one, which includes work from Apple and IBM and Microsoft. The problem with uninspiring style guides is that not everyone needs to take the same information from them.

You know if you’re looking for markup and styles to code up some form of media at component, you’re – probably going to be a techie type. But if you need to understand the balance of type sizes across a typographic hierarchy, you’re more likely to be creative. So what you need from a style guide is different, and yet so many style guides follow the same patterns. The slides are broken by the way. That’s intentionally blank, it’s okay, intentionally blank because I like to with the AV guys so I’m actually going to start off by looking at color, which to me, is one of the most important ingredients in a design because it it communicates enough enough.

It communicates personality and creates mood, and it’s vital to an interactive, understandable vocabulary. Okay, so now it actually has stopped right. No, no! No! We working! So you think right. If we talk about color you’d think that style guides would present color in any number of really imaginative ways right, but you’d be really disappointed because, generally the most aspiring your fine looks like a collection of paint chips from a DIY paint shop.

This is Lonely Planet’s Rizzo. It does a great job of separating design elements from UI components, but you’re going to struggle to get a feeling for Lonely Planet’s design. Just by looking at those color chips. This is green pieces world wide star guide that uses the same approach as the sky’s old web toolkit and the times is functional palette. Goth, UK gothic a we heard of this website. It’s not well known for its creative flair.

It varies the approach by using circles, which I find really strange, because circles don’t feature anywhere else in the branding or the design. On the plus side, though, the designers have provided some context by categorizing colors, like links and text and backgrounds. That kind of thing Google’s material design, offers an embarrassment of colors most helpfully, though it does advise people how to combine these primary and accent colors to into usable palettes, but few style guides actually offer explanations and even less in the way of inspiring examples.

Most are extremely vague when describe color. I love this use color as a presentation element for either decorative purposes or to convey information. The government of Canada’s web experience toolkit states rather obviously, apparently I didn’t know this. We can say this inside Microsoft. I can’t say this inside Adobe, but adding more colors to their palette, as apparently made Adobe a rich, dynamic and multi-dimensional company who knew it.

Was that easy? I’m not sure at all. What makes the draft us web standards guideline colors a distinctly American palate, but it’s going to have to work bloody hard to achieve its goal of communicating warmth and trustworthiness. After a year of Donald Trump, we’re going to build a great style guide, people won’t believe it. It’s going to be so great Mexico is going to pay for it right, the University of Oxford’s, it much more helpful.

They explain how and why to use their colors. They say in a British accent. The dark oxford blue is used primarily in general page furniture such as the backgrounds on the header and footer beautiful. The designers are open table they’ve considered how to explain the hierarchy of the colors that they use by presenting them and their supporting colors in different size chips. So it’s really obvious which are the primary and supporting and accent or neutral colors.

They don’t have to say so Foursquare. They helpfully prescribe the percentages of colors that make up their marketing materials, but there are much more imaginative ways that we can describe color. I love this steel toolmakers brand book, which itself is a really good example of thoughtful graphic design. It could not be clearer when it says use this color with fantastic. You know it presents its information clearly and it’s in a way which is consistent with this no-nonsense.

Steel brand, the designers at Alberta’s corporate identity manual. They cleverly combine colors with imagery of the region that inspires them, and these larger blocks of color give people a stronger feeling for the design without making it difficult to find information about those colors. You know finally, a style guide should demonstrate the interplay of color and typography just to make sure that people understand what’s acceptable in terms of color contrast and accessibility.

So for some companies organizations brand books have developed into a bit of an art form. If you find these things as inspiring as I do, that’s important to remember that we shouldn’t just simply copy the appearance, what we should do is to turn this inspiration into designs that work for the web. You know, for example, this page from the GARP from the Barbican brand book to me suggests things like SVG shapes and CSS blend modes, all that cool stuff.

This is the distinctive McMiller cancer support, visual identity and they’ve brought the design to every page of their brand guidelines. You know the color chips match the style and we get much better understanding of their visual identity when those same colors are used in other kind of imaginative imaginative ways. You know, of course, living style guides and pattern. Libraries. They need to present information about color values, but this is such a fabulous opportunity to be creative.

You know, perhaps we could design some kind of interactive UI that helps people in different audiences get the information they need. This is the Royal Mail, which is one of Britain’s most recognizable brands. I managed to leave brexit island just after Christmas, she’s glad to put on a French accent to get across the border. In fact, when I put on a French accent, they basically asked me to leave, which was fine.

He didn’t mention brexit in the introduction. This was yeah honestly, don’t get me started. I think I got myself started. This is a page from the Royal mallows Brown book. It’s really effective because it brings colour and iconography and typography together into this single page. That gives us an immediate, immediate impression of the brand. You know. What’s more iconic than a van color chips can be used in other kind of imaginative ways to present colour information.

You know they don’t have to be rectangular. If you feel playful shapes with colours, you can better connect them with a brand. This is the designer designers of the the frozen-yogurt. You say, yogurt or yogurt. Here, you’re good yogurt. It is yogurt yogurt, Jurgen fruisé. They did exactly that. What they did was that they don’t just explain their colors. They tastefully display them inside these silhouetted tubs of rose and yogurt, and this conveys personality as well as color information.

You know, style guides can make a bigger impression when they have personality. This is a page from from bings design guidelines and again it brings color and typography graphic styles and their logo together to describe how bings visual identity system is constructed. So I think that style guide should inspire people as well as to inform those who use them, and I was keeping that in mind when I did designed a series of what I call inspired guide.

Design templates – and these are style guide templates that I hope are as beautiful as they are functional, and I made six sets of these things and they all contain pages of design, principles and colors and logos and typography and form elements and buttons all of the stuff That you would normally put into a pattern library or a style, guide and CSS custom properties. I was going to drop some CSS in here at some point.

These now have really really excellent browser support and they’re fabulous for using inside star guides, because they make it so quick to make color changes across different pages that make it so quick and simple. And if you don’t want to use nodejs or npm or any other complicated solutions to get in the way of a of making a style guide, you can just make these things with a good dose of semantic HTML and some plain old CSS SVG, possibly a little Bit of native JavaScript, whatever that is, you know, you should not need a framework, people always say well, you know, can I use this thing with bootstrap? You shouldn’t need a framework or any particular software or toolset to make a living style guide.

You know what we should be doing is removing as many dependencies as we can and if we want to improve the way than which we present Carlow inside style guides. There’s plenty that we can do you know. For a start, we needn’t confine color information just to the color palette, page in the style guide. You know finding imaginative ways to display color across different pages and then show it in context with all the other parts of the design that could be really effective.

This is a six different cover pages that I packed with color. You know they make a really bold statement. You know a visual hierarchy can be easier to understand than just labeling colors as primary or supporting. So we should find creative ways to display that hierarchy. You know you might use pallet panels of different sizes or you might arrange boxes on a modular grid and then fill that whole page with color.

You know don’t limit yourself to rectangular shapes, we can use circles or we can use other shapes that just made with CSS and if irregular shapes are part of our brand, then we can fill SVG silhouettes with CSS and then we can wrap text around them using Css shapes all really fun stuff. You know, for the style guides that I’ve designed I try to go beyond just documenting color and type styles, and I’ve want to describe what they mean visually for the brand.

This was a project that I did last year for Sun Life Insurance in the UK and I described their colors and how to use them across a series of art-directed pages which reflect the kind of lively personality of the Sun Life brand and there’s information about hex And RGB values and SAS variables and when to use these colors for branding or interaction or messaging, you know it’s all there, but it’s in a format that I hope will appeal as much to creative as well as technical people.

You know, and I included information about color contrast, accessibility, because you know understanding why certain combinations of color are inaccessible that can really reduce testing times. You know it can avoid a lot of arguments about color choices, and then I try to find distinctive ways to present color that better reflects the brand. So for sonlife I made these header graphics that really show that their brand personality has a fun side to it.

You know I experimented with ways to communicate color hierarchy through different sizes and quantities of their circle device, so alongside color typography is probably going to be the most defining aspect of any design. So we’re really fortunate now to be able to use any number of different fonts on the web, we’re not limited to using fonts that are just installed on a person’s device. So you might imagine that with so many beautiful typefaces available style guide, designers would constantly be creating inspiring ways to present these types of typefaces.

You know you’d be disappointed. This is Adobe again even Adobe. They own Typekit numerous typefaces, but they fail to make the most from the fabulous fonts in their own star guide. You know most pattern. Libraries offer documentation, but they don’t offer inspiration. You know, while information about type sizes is important, so is the context in which that type is used. The treatment of type different screen widths and the way that type elements interact with each other is vital, so communicating a design.

You know many style guides fail in demonstrating this even the best examples, including this gel from Westpac. It only shows a typefaces basic set of characters. It doesn’t show the context, doesn’t show any other elements. It doesn’t show whitespace. You know as it’s a publication which is written with mostly written content. I think it’d be more inspiring and informative to see examples of actual headings.

This is from this is from The Times of London. Instead of this kind of greeting text, you know people use this quick brown fox jumps over the lazy dog because it contains every letter in the English alphabet, but we can use our imaginations to better demonstrate the characteristics of a typeface. You know over the past few years I’ve been rediscovering the work of accomplished art directors and graphic designers.

You know I, I studied fine art, I didn’t study graphic design, so you know 30, 40 years on this stuff’s all still new. To me. You know learning about this. Work has convinced me that there’s so much more that we can do to communicate our use of type within a style guide. You know looking at inspiration from other medium, including print. We can learn ways to combine color and typography together to better communicate the spirit of a brand and then how to use those assets.

So we’ll go back to the the New York City, Transit Authority, graphic standards manual. You know it immerses the reader in the details of Helvetica more than any publication that I’ve ever seen and it’s showcasing not just the letters but those all-important, often fascinating numerals, and these so often get forgotten when we’re making a star guide. You know when we view characters up close. We can get a much better feel for what gives them their personality in when we illustrate type.

We should also explain how those characteristics of size and thickness and weight should then inform the design of other elements. On a page, including things like symbols, or icons, and understanding how to design typography so that it’s legible and readable, when we present it light on dark, this is something that’s often missing from most of the style guides that I’ve read, as is how typefaces working combination With other icons and their symbols, so this is a page from the New York City, graphic standards, Authority, graphic standards manual, and it just does this fabulous job of demonstrating how helvetica’s, numerals and letters combine to create this design that just feels at home on the new York, metro and style guide should inspire designers by demonstrating other potential applications for typography.

This is the nasa graphic standard manual. It does just that. You know it starts with their iconic logo and I really really enjoy how the designers have explained the background to the typefaces they’ve chosen. You know, I know from experience that including explanations like this can really help people to realize the importance of good type. You know these things are not just commodities that come installed with Windows.

Nasa demonstrates how to use typography on signage vehicle livery. You know, and it’s important to demonstrate how typography can affect even these most kind of mundane items. These are paper forms from NASA, but they could just easily be a an email newsletter or a web form. You know these all benefit from better typographic design and in a style guide. Typography pages can include the examples of the cases and the weights primary and supporting typefaces information about white space.

It should demonstrate how different type treatments should vary. When they’re presented light on dark and as a design often includes supporting and primary typefaces, we should devise ways to show the relationships between typefaces. You know many contain really distinctive character for letters or numerals or symbols. So we ought to make space in the star guide to showcase them. You know, rather than overlook those numbers we should design pages to show when and how and why someone should choose to use numerals for maybe a supporting typeface rather than the primary one.

You know there can be fun. You know we should find ways to display them as that are as attractive as they are useful and to give people a better understanding of how to use them as well as breaking typed down into component parts. You know it’s really important to show the combinations of different type elements, for example in our large images with captions, and we should provide this extra context by demonstrating a type element alongside, let’s say, a typographic scale and then provide information about using color.

At the same time, you know we can use this process of designing a style guide to kind of expand. Our repertoire of designs for little elements that often get forgotten like block, quotes and pull quotes, and we can take the opportunity to bring designers and developers together around a style guide, to talk about the creative possibilities without sacrificing, let’s say, performance or responsiveness.

So when I was designing for WWF, I had to think very very carefully about how to use the typefaces that they’d already chosen. So I started off by making some simple kind of HTML type sheets, and then I tested them across different devices so that I could understand what the minimum and the maximum sizes should be, and then the style guide that I designed for WWF. It includes these organisms. It even includes full page templates so that it gives the fullest kind of impression of my typographic design, and I took a similar approach when I was designing for some life.

This is demonstrating how their bag rounded typeface, how that sort of personality gets conveyed through interactive. You eyes who have included the modular scale, included information about color, and this son lifestyle guide even includes the big larger organisms in a banners and common navigation elements just to ensure that everything is presented as part of a consistent whole. You know to be effective, a style guides, typography shouldn’t just be reduced to its component parts.

We should see typography as an ensemble of type elements that all play their part in a typographic design. Finally, I want to talk about imagery, because graphic icons and graphic images are another important element in the design of a lot of digital products and websites. So let’s have a look at how many style guides routinely display this iconography the Code for America style guide. It presents these icons at different sizes, but the designers at clear left, who we saw earlier they haven’t, explained how the design of these icons should adapt when a get reproduced at different sizes.

You know they’ve also really missed a trick. I think an opportunity to demonstrate adaptability and responsiveness. They could have. You know, built this style guide, for example using picture elements so that we could see what those icons would look like on different device widths. I just I really wish the designers of the map box style guide, had thought just a little bit harder about designing some kind of imaginative interface to make the most of these icons.

I don’t know: where do you have your binoculars, beautiful icons, reproduced, tiny, tiny size and, sadly Atlassian star guide, says absolutely nothing about how to create new icons that match their existing set. This is an important omission. You know. Meanwhile, over at Gavi UK I’ve actually found those circles which we were missing earlier. You know it’s really easy to poke fun. I know that it’s really easy to poke fun, especially at golf dot UK, but it’s really hard to get the design of a star guide right, because designers and developers need something different from them.

You know, developers might need file names and CSS class attribute values. Designers might need to learn how to create new icons with the same visual style as the existing ones. You know, satisfying both needs can be really tricky, but the challenge isn’t any more difficult than the work that we all do every day. You know, I’m convinced that we can do a lot better. You know you might feel more than a hint of nostalgia when you look through British rails corporate identity manual.

Mine arrived late, and now I can only read it. Standing up. British Rail jokes. Don’t go down! Well, in Singapore, I have the rolling in the aisles. If we were in Scunthorpe, you know it’s visual identity, this iconic visual identity extends to the trains. It extends that their uniforms of the people that used to run them very attractive, dapper. Looking chap they’re, you know the best style guides.

Explain the. Why, as well as the how – and this is especially true when it comes to designing things like logo marks or symbols, and I’m fascinated as someone that didn’t study graphic designers to how these symbols are created. You know understanding the thinking that goes into designing a symbol can help people appreciate how and then, when to use very of it. You know this matter is now more than ever before.

I think, because this symbol can be reproduced at any number of different screen sizes across this whole array of devices, and it’s especially important when we think about reproducing symbols against dark color backgrounds, so color, icons and type. They should all complement each other within a design, and style guide should really demonstrate how to combine these elements to achieve the most effective result.

You know, I really love the precision. That’s gone into designing this signage. Is it only me who thinks that this is like the best website button you’ve ever seen in knowing how an icons been designed goes hand in hand with understanding how to use it and fully appreciating principles and proportions makes working an icon into a design at the Right size and with the right amount of white space around it much much easier.

Of course, not every icon or symbol needs precision. I’m always amused. When I read that to maker the Skype cloud, we can use circles any size and placement and shape, but make sure it looks cloud — is –. You know for this style guide that I designed for Sun Life. I went beyond simply documenting the icons and tried to use those icons themselves to add personality to the style guide and with almost all of their style guide pages.

This icon page was, you, know, thoughtfully art directed. I wanted to include the anatomy of an icon and how to design one with the correct proportions. You know. Actually, I adjusted the proportions of the circles in the Sun Life logo so that each one was like 75 % larger than the one before in the sunrise and then built the grid around that. So it explains how I use those proportions to create a grid that then became the foundation for all these future icon designs.

And then I described how to use those same principles to create graphic illustration, so that every graphic illustration, every graphic element, has a relationship back to the proportions in the Sun Life logo and explaining these processes clearly to future designers is really going to help to maintain Standards and hopefully consistency, especially as the collection of these icons and illustrations grows, and I think, because it’s always better to show something.

Rather than tell, I use those same kind of graphic illustrations right the way across the star guide to bring that to life. So in all of the work that I do, I try to use creative design to improve not only how something looks, but also how effective it is and style guides should be much more than just guides to a library of patterns. You know they can inspire people to make better designs, they can teach people the importance of good design and why it’s important to keep our standards high.

You know style guides also the perfect place where we can experiment if you’ve been aching to try out a new design or layout technique like CSS grid. You can do that away from the normal Reince of browser support. You know I want to stretch expectations about what a style guide that’s been developed with HTML and with CSS can look like you know. I want to win, I hope, to inspire people to learn more about those technologies through the style code.

You know it’s also. The perfect perfect place to document the history of a design. These these stories often get lost really easily in the fast-paced environment of today’s design and development. Yes, so, instead of showing finished work in the case, study finished working as a case study in your portfolio. Write about those thought processes show your designs evolution. This is really important to documenting the history of the web.

You know the best the best systems are based on a solid set of principles, so it’s really really important. That style guide explains these well. I promise that I would mention OD again they’ve done what many people haven’t yet achieved and that’s make this tool. That’s as practical for designers as it is for developers. You know, they’ve really succeeded in letting the sole of their brand shine through in the style guide.

I’d like to think we can all learn a lot from that example. Design principles are something that Shopify’s Polaris describes incredibly well, you know hats off to these guys. I think this Polaris is the best style guide. I’ve seen for the last couple of years. It does a lot of things right. You know it starts with accessibility and Polaris makes a really good case for making usability for everyone and building inclusive experiences, part of its principles.

You know it’s showing really what to do and what not to do not just documenting a set of guidelines but showing graphic examples, and I suppose, because a design system is never finished, Polaris also makes it really easy for people to collaborate on its development. You know and why not bring those design principles off the screen and into a three-dimensional world. These posters were designed by an intern, Salesforce and now they’re everywhere around the Salesforce organization, to remind their people of their design principles every day, yeah, you might choose to break the model altogether.

You might combine all of these elements of your visual identity and patterns into a single sheet poster for your designers or developers, or maybe even third-party agencies that you work with to hang up on their wall. You know the possibilities are really endless, so I hope that I’ve inspired you to think a little bit more creatively about the design of your own style guides and component and pattern libraries, and also why creative design can inspire and inform people better.

So thank you very. Very much for listening.

What we’re trying to do is probably next month, we’ll have something which is definitely for beginners like how to make your first CNN model. All of this kind of thing, so next month will be definitely a kind of a more beginner thing. Hopefully, there’ll be tips and tricks which is useful to everyone.

This is build a somewhat beginner, but it’s also kind of a revelation in terms of what’s happened to the natural nature processing thing so so bat meat. Oh, I have a background in machine learning and startups and Finance. I came here in 2013 2014. I was just having fun reading papers and doing robots and drones and stuff too, since 2015, I’ve been kind of serious with natural language processing.

I am a Google developer expert for machine learning. We organized this with Sam I’ve been writing some papers, including with this year. We’ve now got four papers in turnips, so that’s quite good, just small company I mean well, and we also do a developer course. So this is something which I’ll talk about a little bit but red dragon. I think Sam will talk more about this. It’s a Google partner, we work is we quite like to develop prototypes for people, but we’re also very interested in conversational computing, natural voices and knowledge bases, so that’s kind of what we do when we’re not so busy coming.

So I’ve done the Who am I? What I’m going to talk about is a traditional, deep natural language processing now traditional here is. It only goes back about four years five years, because this stuff didn’t exist before 2013 because these techniques were developed then. So when I say traditional I mean not, this is what most of the courses online will teach. You is traditional style. Okay, it’s all changed this year and in particular this summer.

So this is why in-person courses are quite good. So I’ll talk about the traditional thing. I’ll talk about what the innovations have been this year and the new hotness, which is Bert coming from Google there’s some probably no time for actual code, but I’ll give you some hints, okay, so traditional, deep natural language processing. The P elements here will be embeddings super useful, bi-directional LS TM layers, and then we need to talk about initialization and training.

So a traditional model will look something like this. Now, how many people know what I mean when I say: LS TM? Oh, quite a lot. Okay: okay, how many people know exactly what this diagram, or not exactly? How many who have got a good idea? What this does yeah sure it should be pretty much the same three, four? Okay! So basically, this is a fairly standard model in that you put in a word at the book.

These are words going in at the bottom. These are we converted into an embedding. This embedding will then feed into LST m RN going this way and one going backwards as well. These two things will then be added up, and then this will be fed into some kind of classifier, or this is a CRF in a classifier. So this is kind of how people have been doing this for a number of years now and it’s been quite effective.

So clearly it’s been beating the traditional natural language, processing, tf-idf kind of methods which are totally antiquated at that point. So but this is, this is becoming less less useful, now too, so so wordy. Who here has seen about word, embeddings, quite a lot okay, so the idea forwarding reading is just a recap. Is that four four words in text if a word is close to a words in a big corpus, words in a neighborhood should be similar to each other in some way.

So this is kind of a big, a big theme that this this principle should apply to all text right. So what you do to make a word embedding is we assign a vector like an initially random vector to every word, so you can think of this. As being like an Excel spreadsheet and down in the a column, you’d have them every word, so you start with the and of and little further down, you’d have city and further down.

You’d, have you know Space Station as you get rarer and rarer you’re going down the list of words and in English? You’ll have like a couple of hundred thousand words so other across on the rows, though, you’ll have like 300 different numbers, which will represent kind of the vector representation of that word now, in order to figure out what the number should be. You take a window and you slide that across your text and within that window you say: well, these the numbers within this window should kind of average out to indicate each other.

They should kind of predict the words within the window in words outside the window, I’ll actually push away from those. So there’s a way of kind of sampling this so that gradually, as you read this huge corpus, these vectors all get pushed and shuffled around and what happens is, as you give it a decent sized corpus which could be Wikipedia or Wikipedia for starters. Basically, this vector space will self organize and so, in the end, you’ll get a kind of a.

This is a representation from a thing called tensor board, which is part of tensorflow, and basically this can display that essentially the cloud of all words in the English language – and you can also say well, I’m interested in words, which are near say the word important and Because it’s been kind of self-organized, it will know that significance in particular and essential are all pretty close.

So this is a way of essentially for free, just without any linguistic knowledge, apart from giving it a copy of Wikipedia you’ve actually self-organized a whole map of the English language. So what you can do with those word embeddings or that numeric representation, then is put it into an LS TM, which is essentially a network which is the same at every time step, but it passes for at a state each time.

So as it reads this, as it reads the sentence gradually, the state gets more and more involved more and more feature-rich. Now what happens? What one issue with that is that the end of the sentence becomes very feature-rich in the beginning that you hardly know anything. That’s going to happen, which is why you run why these in both directions, so you get features from both ends. One of the problems with this LS TM unit, though, is, if you’re doing a big computation.

You don’t know what what the answer is over here until you’ve calculated one at the beginning, so this kind of forces, as kind of a speed limit on how you can train these things, which is embarrassing if you’re using a TPU which doesn’t want to do this Kind of thing so so unrolling it is forcing the sequential calculation so in terms of initialization, most people will just say well, I’ve got all of this really good stuff in the embedding, and that’s the only free lunch I’ve got right.

Apart from that, my weights in the network are just random: I’m going to use the data for my training tasks to learn the task. Okay and typically, this will use quite a lot of training examples because you’re your network, suppose I’m interested in movie sentiments if I’m just give it a collection of words. It may have a good embedding for each of these words, but it doesn’t know about the syntax of a sentence.

It doesn’t know about how things connect together. Basically, you have to teach your examples of sentiment have to teach it. The English lack the semantics of the English language from the beginning so needs probably a lot of training samples, okay, so this was old-style as of 2013. Let’s talk about the innovations, so there are several things which have come like together and are going to talk to each one of these and there’s a thing called by pairing coding or sentence piece which has been released, open source, there’s transformers.

So, who here knows what a transformer is? Okay, that’s more select crowd; okay, there’s also language modeling tasks which have come onto the come to the for this year, mainly and then there’s also the concepts of fine tuning, which is more like an April thing. So far, right, so what by pairing coding is – and this is this is existed for quite a while, but it’s a technique which is useful because when I mentioned the word embedding before what people do is they have a whole list of every word in the English Language that they can think of or every word in your vocabulary and train it from scratch.

On the other hand, if I came along with a new word, not in the vocabulary, I would just have to assign it an unknown label. I have no idea what the embedding word would be. So if I knew the word for book and the word for booking, but I I wanted to know the word rebooking, I will just have no clue, whereas if I could split the words up into little units of English and English does this quite often rebooking should Be very related to booking and the reconned of does something to the whole booking.

So, if you can, by doing this kind of technique, I can now make a vocabulary, which is infinitely large in the. What I do here is. I would start out with a character encoding, so I’d say: ok, 1 for a 2 for B 3 for C, so I’ve got 26 characters and also I’d have an end of a word character and suppose I actually have a like here’s. My initial vocabulary, where I have low repeated five times, lowest twice newer six times wide.

I can I want to represent this more efficiently. So what I would notice, if I did see counting, is that our plus the end of a word occurs nine times, because this is newer and wider. So I then say: well, our end of word is a new symbol, so at the end of my ABCD up to Zed, I’d have a new symbol called our end of word, so I’ve just done emerged and I’ve now replaced all the art in the woods. With this new symbol, which could be fire or something – and I could do this again and again so it you know – er would be repeated nine times or E or W of replacing nine times so and I’ve got an earth sound okay.

So this is kind of a useful English in you could imagine also that the in a big corpus thee would be taken into a single word very quickly off a single symbol very quickly. So if I just do four steps of merging basically, this would allow me to take an out of her company word like lower and it’ll just be composed of low and she’s kind of what you’d want, and what people are found is that this works really well.

For English language – and so this, this enables you to build an infinitely large dictionary just out of the components which arise naturally – and you can learn this without any linguistic knowledge, because now now you all understand how it works, and you can also read the paper and Download the code from github, so this this sentence, peace thing which implements this technique and a couple of others is this – is a good way of creating sorry.

All of these slides will be on the meetup link. So, underneath the meetup discussion, all the slides are going to be there. So I welcome the pictures, but you know okay, so a sentence peace is released. Now the next thing is a transformer now this is a kind of this is for a single sentence piece. We would then do something with attention written in the box and then various norms and free for words and then at the end, we’d have some kind of protection, so this is a purely feed-forward kind of network and in these papers you typically do twelve sets of These do quite a large pile of these things.

These are all based on CNN’s rather than Ireland’s. So when you get to a proper model, basically you’ll be taking in the words at the bottom, with some embedding. You then have this thing, which is a tension, so, instead of an RNN where you’re rolling forwards the whole time, the attention is looking simultaneously, at all words in the sentence to find out what is the most relevant word. So if i’m talking about or going to the bank to make a check deposit, the fact that the word deposit does depend on word, bank and the sense of bank depends on word check.

Okay or it’s not really a linear relationship. These have, basically all of these words, will kind of vote for being picked, and so this multi this insane is for flows up through this model, and essentially it’s is extremely powerful. Where doing in so there’s a paper called attention is all you need where they pro excellent results, just using this attention mechanism without any are and ends, or anything like this.

This is a pure, CN n method. Now the other trick that people have cottoned onto is basically like embeddings we’re going to train on a huge corpus of text, but this time we’re interested in getting the context a bit more solid. So what we all do is we train the entire network instead of just training a vector at a time we’ll train the entire model it at a time over the entire text and with kind of things that we’ll do is get the model to predict.

Just one more word so I’ll give it a sentence. You know I went to the bank to deposit a and I expect it to tell me check and if it doesn’t I say no you’re wrong. Tell me you’re a check, and then I say then what’s the next word is probably for stop. Next, word is otherwise I might run out of money or so, though, they’re basically there B the whole basis whole set of things. You might say after this and a whole lot of words, you would not say so.

By doing this, you can you just use a corpus of text to roll out an understanding of the English language in a fully unsupervised way. So, instead of just doing these vectors, you’ve trained the entire model as well as the embeddings at the beginning, so that was out that this predict. The next word was a language task. There’s another one called a closed task to her. Basically, you take a sentence, and you start deleting words say it fill in the blanks.

Another thing you might do is you could take two sentences and switch between them and say tell me where the switching point places. So if I started with a sentence about one thing and switched to another thing, you could say: okay, if you can identify that is then you understand language better. So, just all of these kind of tricks are basically forcing it to understand language more and more, and you can do it almost for free.

Another thing you could have is a novels okay, so you could have a sentence which is comes from one part of the novel and follow it with a sentence from a different novel. Now, if you can detect – which ones which like, is this a decent Sikh decent sequence of sentences or not, you can actually get sense across large passages of text, so these things can be trained almost for free in terms of linguistic knowledge, it’s kind of obvious that This this is a thing, but people weren’t doing this quite a while ago, or a year ago, people weren’t playing this game.

So much so another thing which came along – and this is partly due to faster ie them hustling. These models is, you can take a model which is pre trained on a large corpus which will be that a language model for a predict, the next word kind of model, and then, if I’ve got a say, I want to do movie sentiment. I would take a whole bunch of movie reviews, but even if I didn’t know whether they were good or bad, I just take as many reviews as I could find anywhere and then just train the language model to understand movie review language more.

So I don’t I’m not telling it the good or bad yet I just wanted to understand what people say about movies and then at the very last moment I say: okay, I’ve got a hundred movie review. Examples learn whether learn to say good or bad and response to each one. Now this approach can work much better than the previous approach, which is train from scratch, where you might need 10,000 movie reviews, because you’ve got to learn a lot of the movie review language simultaneously.

Here, these models have got a very good understanding, a very good measure of English language now you’re just trying to train it on just the fine tune, the last a little bit. So this is a super effective technique, and this is why these things are getting very good. So so, in terms of recent progress, you’ve got in February 2018. We’ve got a thing called Elmo owl fit came in in March. In sorry May 2018 open ie.

I came up with this transform or model in June 2018, and now the new hotness is Bert. Right now, Bert is clearly the successor to Elmo, though I don’t think they mention that so this was done in October, so this is fairly new and so what you do with Bert. Basically this is this. Is one of these models full of Transformers? Do you do the the sentence piece thing at the beginning? You have this whole transformer thing and then basically we need to train it to do lots of different tasks.

So one of the tasks could be this, which is a squad squad, so basically to do the squad. The squad thing is, I ask a question and then sorry, I have a question and I need a response and so basically the question will be. You know when was the beginning of World War two, and now I would give it the Wikipedia article about World War two, and I would know the extent of when, where the answer was in terms of words, and so the answer for squad is tell me what The start word and the end word, and so in order to train this task.

Basically, you put in the question, have a separator put in the passage and then just force it to tell you I’m a start and I’m an end. So this, basically all of these kind of common language tasks can be forced into the same model so rather than build special models for every different task. You can kind of say I want this to perform in this way and use choose the standard model, and this is way too small.

But basically – and I would encourage people who are interested to read the book paper because it’s actually quite well-written and quite it lays it out nicely. So basically, you’ve got the previous state of the Arts at the top and then so. You’ve also heard about the by Alice yen thing here. This is the open AI thing and basically, these scores are very much higher than they used to be. People were slowly climbing up.

So if we look back in history, we’d have had the old approaches which should have been scoring. You know, say: 50 when you got to the deep learning stuff in order jump to 60 now people are clawing their way up and now we’re at 70. I mean that that’s the kind of the impression to get to so this Bert performance suddenly has been a sea change this year. These states of the arts have been beaten by wide margins as we go through the year.

So, what’s nice is that Burt has got working code on github, okay, it’s a patchy to license. So anyone can use this wherever you are. They have scripts to produce all the results in the paper. They’ve released, pre train models, both regular sized and large size. So large is really for the TPU people, regular than you need kind of like, as you know, twelve gig GPU to use this thing. So this is not a light model and they’ve also released in both regular and large english-language model, a multi-language model which does 102 languages simultaneously and Chinese.

So there’s kind of they’ve been hard at work with their TPU pods and they’ve pre-trained all this stuff. We know that it costs to produce one of these models. It’s a very large amount of money. On the other hand, to fine-tune one of these, the fine-tuning can take place in in under an hour or five minutes, depending on the size of your tasks. So the fine-tuning stage can be very fast compared to running this and building this thing initially on Google size data, the other nice thing is they’ve got a ready to run collab.

So I won’t do it now, but you could click on this and get just mint at the turn. You get the whole thing with how to run this thing on the cloud TPU or this is a machine which has got GPUs already. This is all for free, so collab has got TP used for free, go up a TPU for free. You can just run this and have a play with it, so that that’s kind of interesting when it works. Oh, so let me so if you’ve got a problem, the old way of doing this, would you be you build your model? You take a glove in bedding which is kind of one of the freely available models freely available.

Beddings on that the Internet. You train it up. You need lots of data. The new method. Is you, take a pre train, Bert kind of model, you’d fine-tune it on all the unlabeled data you could ever find. Maybe it’s emails. You just put in you dump in your Gmail or whatever, and then you put in the few emails that you care about. You need far fewer of these than you would in a regular model. You don’t need as much data and you should expect better results.

So, as a wrap up, Bert is kind of the latest innovation in this kind of NLP trend, which has only been taking place kind of this year. It’s like beaten all the state-of-the-art performance benchmarks. It’s fully released with all the results just out in the open, as code should be, and people are describing. This is kind of the imagenet moment for NLP in as much as for imagenet people train this the thousands thousand class image models until they were better than humans.

Basically, but what people didn’t realize or didn’t pick up on so quickly was that the transfer learning of these models once you have a superb image net model, you can then use that to recognize images on mammograms of you know all sorts of different things. You’ve built a very good image vision model, even though you’ve trained it on dogs, mainly you’ve got an excellent model which can be applied to lots of tasks and it’s all released for free, and so this is kind of a very similar kind of moment.

Um. Ok, so that’s my talk, um. I will have questions in a little bit. Oh I’ll do a couple of adverts, though, because I’m that guy so you’ve already found the deep learning meetup group because you’re here – and we have the next one in December – here – probably typical to onset. So it talks for people stand starting out. Hopefully, if you’re a beginner, you found something interesting in what I said, but we’re next month in particular we’re going to focus on having something good, something from the bleeding edge.

This is fairly bleeding edge as well, so because it’s frightening – and hopefully we can get some lightning talks. So if you want to give a five minute talk about anything at all related to this kind of suffering, just have a word with at the end, and also we now figure out, we’ve got over three thousand members in this group, making it one of the largest Tensorflow groups on earth, which is country all to you right, um, so Sam and I run deep learning workshops.

So the we call these jumpstart workshops, there’s a there’s other courses beyond that for more advanced stuff. But basically, what we’re trying to do here is for two days and there’s online content as well. We force people to actually do stuff on their laptops and also do some kind of project of their own, devising or or one of our kind of sample projects. If you want to – but this is, we feel that the hands-on thing is super important, because just clicking through some, you know some pre-made modules.

It is not really so helpful these if you’re a Singapore or P, aha, Singapore, citizen or PR. This cost is heavily subsidised up to apparently up to a hundred percent find out if you’re, one of those lucky people. By going to this SG innovate thing and if you’ve got to the right place on SG innovate, it will look like this. Okay that were okay. There’s a little about the the deep learning developer course.

This is a full course when we ran one of these last year, quite there, a number of people with some some people are even swapping jobs after the better job they achieved. After doing the courses like awesome return on capital there, so that this is something that dates has yet or unknown, but there’s definitely more to come after the jump start. But what we want people to do is do the jump start before the full thing.

Just so that they know how to run these models, and we don’t have to explain everything from the beginning again. Every time and one final thing, okay, is I we have a little company Sam and I have a little company called red dragon that we are on active kind of intern intern hunt. We are looking for people who want to do deep, learning stuff as much as possible, though we recognize that people have got academic obligations.

We have had someone over the last couple of months working with us and he’s now appearing at nips. So that’s kind of an interesting thing for a Singaporean student at the moment, so we kind of said when we originally said intern hunt that this was a possibility, and this is an actuality. So we’re very grateful to him. But please cut approaches if you’re such an intern: okay, we’re up next with that, guy that guy okay,