I am known as the automation Panda. I love software testing, automation, behavior, driven development and Python. You can follow me on Twitter and my blog at automation, Panda. I like to thank my company for sending me here today. I work at precision lender I’ve been there for just a year as of this week.
Precision lender does applied banking insights and a couple of us will be at PyCon this coming May. So, thank you for sitting lender. Here’s a question for everybody. Raise your hand if you have ever hit a software bug everybody to put your hands up. Okay. Thank you. Great we all we all have experienced a fair bugs. Do we like software, bugs no they’re bad, thus far, they’re bugs that’s why we want to squish them so here’s another question.
Raise your hand if you always test your code, Wow Wow yeah, that’s about what I expect I’ll, give them a sulk a couple times and that’s usually what happens now, I’m not here to do shaming or anything. It’s not bad. There’s there’s reasons it’s okay, but I want us to stop and pause for a moment think time out what why is there just this discrepancy? We know we have all these software bugs, but we’re not always testing our code.
Well. Why is that and, furthermore, why don’t we automate the tests so that they test the code gets tested automatically, so we can spend more time making the code to begin with. This is not an easy question to answer, because there are some tough points we have to face. First of all, especially with test automation like we know deep down, this is something we should do right for small projects. It’s not such a big deal, but for big projects, things at scale, things that are have huge teams working on them.
We have to move fast. We have to move continuously and there’s lots of moving parts as there’s lots of danger areas that things could just break and all those bugs just rush in like cockroaches right. We know that we also don’t have enough time in the day or even in the universe, to test everything manually right. We can’t repeat the full testing of these giant systems over and over again ourselves, there’s just not enough time for that.
We need some solution to do things automatically and that’s where test automation is supposed to help us. Unfortunately, though, we also know the test, automation is hard and that’s one of the barriers of entry of why I think so. Many people didn’t quite raise their hand. When I asked do you always test your code test automation, it’s not an easy thing. I do this day to day. I know firsthand, it’s not easy.
I fight the problems in the test automation every day. It requires strong developer skills, it has its own technologies, it has its own software stack just because you’re, a good developer doesn’t mean you’re necessarily going to be a good test. Automation engineer, because there are certain aspects to that domain that are tricky, and I could. I could spend all day talking about that, but we ultimately, what I want you to recognize is it test.
Automation is not just about writing test scripts test. Automation itself is software and deserves same practices and tools and attention that we give the product code so to speak, and with that in mind, I’ve also from talking a lot of people, different companies and teams. I found that test automation can be intimidating for a lot of folks. Many teams really don’t know how to start. They know they need it.
They know they have bugs, but they don’t know how to to to approach the test. Automation challenge so to speak, and so I see two things happen: either teams will keep punching the football down the core and just keep avoiding test automation altogether and they’ll either skip test coverage or the higher teams just to do manual testing, and that’s not a really Good situation, because that’s not scalable or a lot of times what happens is they’ll try, but because they don’t train themselves appropriately, they don’t get the information they need, they don’t really try or they don’t take the the careful steps to begin with next thing.
You know you’re later they’ve got thousands of tests that are failing every night and they’re not getting value out of them, and they just want to nuke the whole thing and say we wasted time. Let’s start over right: that’s not good, both of those situations not having anything or having stuff. That is detracting from your value very bad, and we hit this moment where we began. What do we do with this? Because we know we need this, but we’re not doing it well and that’s what I’m here to talk about today and in fact I’ve got a little story too to go along with how we can approach the test, automation challenge.
Now I want you to meet a friend of mine, let’s meet someone who can help I’d like you to meet my friend Amanda. The Panda amanda is a software engineer who loves using Python. Amanda works at a company called bamboozled. Yep bamboozle is a small but growing tech company bamboozle provides an online platform for panda commerce as a full-stack developer. Amanda has a good understanding of bamboozles web front-end REST API service layer and infrastructure needs recently bamboozled had a couple severe bugs escaped to production.
Oops egad management decided that was finally time to start automating tests and Amanda volunteered to lead the effort. Since this is the first time she’s done test, automation, Amanda does her homework. She starts learning about different testing types tools and frameworks. However, her web research feels more like a game of buzzword bingo, because there are so many buzzwords when it comes to testing and automation, and everyone seems to give different definitions to the same words.
Anyone have that problem before true story, so like what are the differences between unit integration and to end testing we’re all the packages. I need with this cucumber thing. The selenium webdriver thing: do they work together? They separate from egad, we don’t know, we need help. So Amanda calls her friend Andy. That’s me, software engineer and test there’s so much info out there. Amanda says that I can’t discern what is the right approach to take.
We want to do things right. How do we do it right? Let me ask you a question. Andy says: what is your main goal with testing? What is your main goal? Well, Amanda replies. I guess I want to make sure that the critical parts of the bamboozles app are working correctly. Any time someone introduces a bug test should catch it immediately great. It sounds like you need a functional test, automation, solution, functional test to make sure the features work correctly.
As you said, functional tests serve two main purposes: validating goodness and identifying badness I’d like to share with you to test automation pyramid. Has anybody seen this before some people? Okay, cool? Let’s talk about this, the pyramid is a rough guideline for how to automate how an automation solution should be structured. All layers are functional tests from unit to integration to end-to-end at the bottom, we have unit tests which cover individual units of code like functions, methods and classes.
Unit tests are white box in that they interact with the product code directly they’re, making calls to your functions and classes above that are what we call feature tests which cover blackbox behaviors of real live instances of product under test. So you don’t directly access the code. You access whatever the deployed version of that product is integration, tests cover components or things needs usually like a want to hop.
Maybe a service API and to end tests come a pass through the entire system. Typically, like a user would exercise through the web UI, each layer of the pyramid mitigates risk at its optimal return on investment. That’s why we have the layers. We want a strong foundation of unit tests because they are small, quick, easy to maintain and will catch things very quickly. Integration and especially end-to-end tests are far more costly to build a maintain and they also take more time to run.
I want you all to consider the rule of laws. A unit test would take about one millisecond to run an integration test. Usually it would take about one second to run and an end-to-end test order of magnitude will take about one minute to run so there’s this issue of scale as we go up the pyramid hold on, Annie, Amman Energex. Why do we even need feature tests? They sound like such a burden when compared to unit tests.
I also read that web tests can be flaky with selenium-webdriver Andy replies well hold on now unit tests don’t catch all problems. Here’s an example: back in 1999, NASA lost the Mars climate orbiter when it disintegrated into the Martian atmosphere, h.Y ground-based software did not convert US measurements into the metric system. This is a real story. The calculation modules worked individually, but they did not integrate properly.
Had NASA checked software with proper integration tests, they would not have lost six hundred fifty five million dollars. Wow Amanda explains. I wouldn’t want that to happen to my company. It sounds like I need to take a pyramid approach, so what skills do I need to get started? Mandy smiles, good news. You already have the skills test, automation is software and the same development principles apply just pick a core framework and get rolling.
You may also want to take some online courses about frameworks and test packages that you’ll need check out test automation University, which offers a bunch of courses from testing experts for free, so there’s another question: Amanda asks what test framework should we use? Bamboozle does a lot of Python on the back end, so I looked at frameworks like unit tests, PI test and behave. Python is one of the best languages for test automation and E replies, and you can’t go wrong with PI test.
Pi tests can handle tests at all layers of the pyramid, markers can categorize tests, test functions and fixtures of lightweight and plugins can be added for reports, code coverage and even parallel execution among all Python test frameworks. Pi test is also arguably the most pythonic, though that could be up to debate now. A test automation solution is more than a score framework, though so be cautioned successful automation projects bring together readily available packages, rather than developing everything from scratch.
For example, you may want to use Python standard logging library for logging messages. Requests would be great for testing rest. Ap is selenium, webdriver and splinter would be great for automating web UI interactions, and there are also tons of PI test plug-ins for different frameworks like flask or Django, so be sure to look before you leave to see what packages will be useful for you. I can also recommend a few books to get you started.
I’ts Python testing, with pod tests by Brian Aachen, is an excellent resource on the PI test framework. Also by Bruno Oliveira is high test Quick Start Guide, which is another take on how to get started with PI testing and for general automation. Concerns in Python, hands-on Enterprise, automation, Python by bass. O’ralley is another great resource, so check these books out. If you want to learn more creative, I sandy Amanda says this will help me get started.
So Amanda goes back to the imbecile. It starts ready tests in Python using PI tests. She found it was fairly easy to get started even though she’s new, with test automation. Taking a pyramid approach, she starts by beefing up the existing unit tests and then she moves on to using requests for REST, API testing and selenium webdriver for web UI testing. She puts code in the same repository as the product code so that they can be version controlled together.
However, after writing, a number of tests Amanda discovers some frustrations. She meets up with Andy for bubble tea. To pick his mind, Andy says: how do I avoid duplicating so much test code? So many of these test steps are the same. It’s very repetitive. I just need to make minor tweaks here and there code duplication is code, cancer and it replies. Have you heard about property testing with hypothesis? It could help you with your unit testing.
I thought that this hypothesis automatically runs tests against a wide range of scenarios, so you don’t need to hard code variations also to avoid duplication with feature level tests. Try using gherkin with a PI test, BDD plugin, you can write test scenarios in plain language using given length in scenarios and those will be glued to Python functions that will execute like a test script. Steps can be weary.
Steps can be reused by different scenarios and writing in gherkin. First helps you mentally separate test cases from test code, so you can focus on writing good tests. I’ll definitely check out hypothesis and BD high test. Bdd Amanda says I think there were talks about them at PyCon, 2018. Now here’s another question: how have you handled test data? In the past? My unit tests use PI tests monkey patch fixture to mock data, but I’m struggling to find good ways to manage data for bamboozled systems for black box testing test data is definitely hard to maintain.
Try to keep test data to a minimum for most tests. Data needs are actually pretty small tests should set up and clean up whenever they need, such as database records or custom settings. Some data, however, like user accounts, are expensive to prepare and should be handled outside of the tests and treated as a ready State. It may also be good to periodically set up and regenerate that data to make sure your systems are clean after any unexpected damage could happen.
That makes sense. Samantha says I’ve also been storing inputs like URLs and usernames and local config files, so that they aren’t hard-coded into the test code. Awesome Andy says that’s great. You may also want to consider using some sort of key management service like AWS kms or as your keyboard. That way your passwords will be secure and your config files will not go. Stale Amanda goes back to bamboozle again and she uses what she learned to improve.
Her test automation project, it becomes her full-time work and she transitions unit tests work to other developers, so she can pose focus exclusively on future tests. She does so well that her team keeps asking for more and more tests, despite training other developers, to help write tests. She becomes buried by the backlog. Amanda takes a step back and asks herself what tests should we automate? Her answer is simple.
Automation is expensive, so focus on the highest return on investment. Amanda writes guidelines for her team to avoid getting crushed by an impractical 100 % test automation goal. She recommends automating, happy paths of core features that will be run repeatedly. She also recommends avoiding edge cases, one-off checks and visuals, because those would be much easier for an expert to log in and try manually.
Amanda also recognizes that these are simply guidelines for her team and her project at bamboozle, and they may not necessarily apply universally to all other circumstances. However, even after narrowing the scope of the test to automate the number of tests is still quite large. Amanda’s team has hundreds of future tests that will balloon to thousands in the upcoming year. The service level tests now take about ten minutes to run and web tests take over an hour.
It’s no longer practical for developers to run the tests on their local machine. So Amanda calls a meeting with her friend Auto the Panda another developer at bamboozle and she asks how can we reduce the burden of automating tests or how can we reduce the burden that automating tests puts on our development team? How can we reduce this burden? Auto says it would be really nice if we could run these tests as part of a continuous integration and delivery pipeline.
Then every code change I make as a developer would become automatically tested once it gets pushed. The only problem we still have, then, would be that the tests take a long time to run what if we run the tests in parallel using PI tests, X’test Amman ax asks it could plug directly into our existing framework and parallel testing. Even though we’ll be hard on our laptops would be much easier to scale in a CI environment with shared compute resources, I can make sure that the test code is thread safe and would have no collisions such as web logins and thankfully PI test fixtures can help Us set up and tear down everything cleanly and safely.
If necessary, we could even set up selenium grid to scale our test, automation, execution across multiple nodes, with multiple platforms and multiple browser versions, or we could also look into platform as a service providers such as sauce labs or browser stack auto-replies. That would be awesome. We also need to make sure feature owners are automatically notified of test failures that way they can triage any failures immediately and we get that vast feedback cycle Amanda and Otto team up to make it happen.
Amanda updates the automation code to use pipe test X, test and Otto adds new jobs to their existing Jenkins. Build server for running tests means they also set up a report portal instance to to their test results. Dash board tests now complete in a fraction of the time and developers get failure. Notifications immediately, unfortunately, though, adopting true CI brings unforeseen consequences of scale.
More test runs means more issues are uncovered, while most test runs are fine, some start failing intermittently ouch, and that is one of the classic problems with test automation. There are four common reasons. We see these test failures. It could be that the feature under test has a really tricky bug. It could be. The feature under test was the feature itself was updated without also updating the test code.
It could be the automation didn’t, handle an unexpected condition, such as a timeout or egad. The worst, the automation itself has a buck. None of these, though, are good outcomes. Unfortunately, as well. The failures leave a bad impression on the team. Some of Amanda’s co-workers stopped triaging test failures and they even suggest skipping the tests altogether. A man who doubles down. How can we make tests more reliable? She asks herself first.
She realizes that continuous integration is the production environment for tests and should be treated as such. I’r going to repeat that again: ci is production for test automation. Test code should undergo the same best practices as product code, including coding, standards and review share test framework components should also have unit tests because they’re shared by multiple things. Secondly, Amanda discovers that the most common points of failure are remote calls and race conditions.
She Tunes the framework to have appropriate, timeouts, immediate retries and helpful failure messages when problems arise, she makes it correct what it can and safely abort when it cannot. She also looks into something called contract testing, which uses mocks for services to avoid intermittent Network problems. Third Amanda reaches out to her team members for better collaboration because it starts with the team.
The Institut better processes for flagging feature changes that will affect tests and also for being gracious when failures happen on both sides. As a result of these three improvements, automation becomes more reliable and confidence in the whole test. Solution has been restored by this point. In our story, the automation solution is quite mature. New tests are added every sprints and results provide fast feedback to the development team.
Other developers like auto, join Amanda in automating tests at all layers of pyramid as part of their natural development. Workflow quality becomes centric to the team’s vision, and that is Amanda’s test automation journey. There’s so much more. I could say about all of this. A half-hour slot is just not enough time, but I do want to share some resources if this talk was meaningful to you and you want to go, learn some more.
This is the slide to take the pictures of it’s just disclaimer boom right. If you want to learn more, I highly recommend my blog at automation – Panda, yes, I’m biased towards that one, because it’s mine, I also recommend, checking out test automation University as well as the Ministry of testing the three books and their authors are listed here as well. Those are the ones I mentioned earlier in the slides and at the bottom.
I’ve also listed all different frameworks and packages and such that I recommend it as part of Amanda’s journey, so you can learn more about those and so yeah. Thank you so much for inviting me here to Texas. This is my second time in Austin. If you have any any other questions or want to talk more, please come see me. You know whenever I’ll be here both days, you can reach out to me on Twitter. At automation, Panda, you can contact me through my blog.
Thank you. So much appreciated.
Don't have time to do the blogging thing?
Maybe Copywriting services are for you.
Jimmy's Internet Marketing Business Blog 