Difference between pages "Arrivals Departures" and "DevTestCIminiconf"

From LCA2015 Delegate wiki
(Difference between pages)
Jump to: navigation, search
 
 
Line 1: Line 1:
The intent of this page is to allow conf organisers to know who's expected at the airport when.
+
== Developer, Testing, Release and Continuous Integration Automation Miniconf 2015 ==
  
You might also be able to use the info below to connect up with people who are on the same flight as you, arrange taxi shares, and so on, so '''leave some contact info''' in the entries that you create. Remember, there is '''no free transport from the airport''' provided by the conference this year. See [[Getting_to_Auckland#Airport_to_venue_transfers|Airport to venue transfers]] for more information.
+
This miniconf is all about improving the way we produce, collaborate, test and release software.
  
===Arrivals===
+
We want to cover tools and techniques to improve the way we work together to produce higher quality software:
  
Order the entries below by scheduled '''time of arrival''' in Auckland. ''Please use 24 hour clock format (hh:mm)'' and ''Auckland time!''
+
* code review tools and techniques (e.g. gerrit)
 +
* continuous integration tools (e.g. jenkins)
 +
* CI techniques (e.g. gated trunk, zuul)
 +
* testing tools and techniques (e.g. subunit, fuzz testing tools)
 +
* release tools and techniques: daily builds, interacting with distributions, ensuring you test the software that you ship.
 +
* applying CI in your workplace/project
  
==== Tuesday 6 Jan ====
 
* 07:50 JQ244 (From CHC)
 
** [[User:Andrew Sands|Andrew Sands]]
 
** Lisa Sands
 
* 14:35 QF8770/EK434 (BNE to AKL)
 
** James Iseppi
 
** Lee Symes
 
** Matt Franklin
 
* 23:35 JQ205 (SYD to AKL)
 
** [[User:Arkady Gundroff|Arkady Gundroff]]
 
  
==== Wednesday 7 Jan ====
+
== Schedule ==
* Afternoon/Evening Driving from Wellington
+
** [[User:Andrew Ruthven|Andrew Ruthven]], Susanne Ruthven and kids
+
* 10:40 Nick Coghlan - [[#Beaker's Hardware Inventory System|Beaker's Hardware Inventory System]]
* 22:05 QF147 (from Sydney)
+
* 11:10 Steve Kowalik - [[#Testing the cloud in the cloud|Testing the cloud in the cloud]]
** Steve Walsh
+
* 11:35 Fraser Tweedale - [[#The Best Test Data is Random Test Data|The Best Test Data is Random Test Data]]
 +
* 12:00 Sarah Kowalik & Jesse Reynolds - [[#Developers, sysadmins, and everyone else: Why you should be using Serverspec|Developers, sysadmins, and everyone else: Why you should be using Serverspec]]
 +
* 13:20 Matthew Treinish - [[#Subunit2SQL: Tracking Individual Test Results in OpenStack's CI System|Subunit2SQL: Tracking Individual Test Results in OpenStack's CI System]]
 +
* 13:45 Raghavendra Prabhu - [[#Corpus collapsum:  Partition tolerance of Galera put to test|Corpus collapsum:  Partition tolerance of Galera put to test]]
 +
* 14:15 Sven Dowideit - [[#Kickstart new developers using Docker|Kickstart new developers using Docker]]
 +
* 14:35 Joe Gordon - [[#Large Scale Identification of Race Conditions (In OpenStack CI)|Large Scale Identification of Race Conditions (In OpenStack CI)]]
 +
* 15:40 Anita Kuno - [[#Gerrit & Gertty: A Daily Habit|Gerrit & Gertty: A Daily Habit]]
 +
* 16:10 Dr. Jason Cohen - [[#Incorporating the Security Development Life Cycle and Static Code Analysis into our Everyday Development Lives: An Overview of Theory and Techniques.|Incorporating the Security Development Life Cycle and Static Code Analysis into our Everyday Development Lives: An Overview of Theory and Techniques.]]
 +
* 16:35 Discussion / Q&A
  
==== Thursday 8 Jan ====
 
* 22:05 QF147 (SYD to AKL)
 
** Hugh Blemings
 
** Rachael Blemings
 
  
==== Friday 9 Jan ====
+
=== Beaker's Hardware Inventory System ===
* 06:30 NZ176 (from Perth)
+
** [mailto:me@kye.id.au Kye Russell]
+
** Delan Azabani
+
** Brock York
+
** Josh Batchelor
+
** Luke Mercuri
+
** Clayton Johnson
+
** Matthew Pen
+
* 15:30 NZ 5 (LAX to AKL)
+
** [http://alasdairallan.com Alasdair Allan]
+
* 20:50 VA7445 (SYD to AKL)
+
** James 'Ender' Brown
+
* 22:45 VA7434 / NZ 134 (BNE to AKL)
+
** Clinton Roy
+
** Anthony Towns
+
* 23:00 VA 716  / NZ ??? (SYD to AKL)
+
** [[User:Craige McWhirter|Craige McWhirter]]
+
  
==== Saturday 10 Jan ====
+
By Nick Coghlan
* 00:05 MH-133 (KUL to AKL)
+
** [[User:Raghavendra Prabhu|Raghavendra Prabhu]]
+
* 05:45 NZ3 (LAX to AKL)
+
** [mailto:bdale@gag.com Bdale Garbee]
+
** [mailto:keithp@keithp.com Keith Packard]
+
* 08:00 QF111 (PER to AKL)
+
** [mailto:kibelan@gmail.com Ben Kelly]
+
* 14:35 EK434 (BNE to AKL)
+
** [http://www.humbug.org.au/RussellStuart Russell Stuart]
+
* 17:15 NZ124 (MEL to AKL)
+
** [[User:Christopher Neugebauer|Christopher Neugebauer]]
+
* 17:30 NZ136 (BNE to AKL)
+
** [https://twitter.com/bradleymarshall Brad Marshall]
+
* 18:20 NZ792 (ADL to AKL)
+
** [[User:Daniel Sobey|Daniel Sobey]]
+
* 18:55 QF181 (SYD to AKL)
+
** [mailto:lca@eyal.emu.id.au Eyal Lebedinsky]
+
** [[User:Jonathan Woithe|Jonathan Woithe]] (from QF738 departing ADL)
+
* 20:50 VA7418 (PER to AKL)
+
** Byron Jones (glob)
+
* 22:05  QF147 (SYD to AKL)
+
** Jason Lewis (@jasonblewis)
+
  
==== Sunday 11 Jan ====
+
Ever wondered what it might take to track down and
* 00:45 VA160 (BNE to AKL)
+
resolve a kernel bug that only affects one particular variant of one
** [https://twitter.com/gm_stack Geordie Millar] (from VA1393 departing ADL)
+
particular processor architecture from one particular vendor? Step one
** Michael Wheeler (from VA1712 departing GLT)
+
is going to be actually getting hold of a suitable machine, and for
* 05:45 JQ215 (MEL to AKL)
+
that, you'll need an inventory system that provides that level of
** Michael Cordover (aka mjec)
+
detail.
* 07:00 NZ406 (WLG to AKL)
+
** [[User:James Forman|James Forman]]
+
* 07:10 NZ15 (SFO to AKL)
+
** John Dickinson
+
* 08:00 QF111 (PER to AKL)
+
** [[User:Andrew Buckeridge|Andrew Buckeridge]]
+
** [mailto:russells@adelie.cx Russell Steicke]
+
* 10:10 NZ508 (CHC to AKL)
+
** [[User:Leroy Hopson|Leroy Hopson]]
+
* 13:05 NZ4994 (HKG to AKL)
+
** Jussi Pakkanen (from LH730 departing MUC)
+
* 13:45 QF8762 (also EK406) (MEL to AKL)
+
** George Patterson
+
** Kevin Collas-Arundell (@kcollasarundell|tacticus)
+
* 14:00 EK0412 (SYD to AKL)
+
** [mailto:diego@biurrun.de Diego Biurrun]
+
** [mailto:danbryan@gmail.com Daniel Bryan]
+
* 14:15 NZ454 (WLG to AKL)
+
** [[User:Jeremy Visser|Jeremy Visser]]
+
** [[User:Ewen McNeill|Ewen McNeill]]
+
* 14:25 VA7422 (MEL to AKL)
+
** [mailto:jon@oxer.com.au Jonathan Oxer]
+
* 14:35 QF8770/EK434 (BNE to AKL)
+
** Jared Ring
+
* 14:50 NZ102 (SYD to AKL)
+
** [mailto:himangi774@gmail.com Himangi Saraogi]
+
* 14:50 VA934 (OOL to AKL)
+
** [[User:Katie Miller|Katie Miller]]
+
* 14:55 QF143 (SYD to AKL)
+
** [mailto:mike.carden@gmail.com Mike Carden]
+
** [mailto:remi@remlab.net Rémi Denis-Courmont] (from QF2 departing LHR via DXB)
+
* 15:10 QF123 (BNE to AKL)
+
** [[User:Mark Ellem|Mark Ellem]]
+
* 15:25 VA148 (MEL to AKL)
+
** [https://twitter.com/fukawi2 Phillip Smith]
+
** [https://twitter.com/smarthall Daniel Hall]
+
** [[User:Tim Serong|Tim Serong]] ([https://twitter.com/tserong @tserong])
+
** [[User:Stewart_Smith|Stewart Smith]] ([https://twitter.com/stewartsmith @stewartsmith])
+
** [[User:Jack_Scott|Jack Scott]] ([https://twitter.com/JackScottAU @JackScottAU])
+
* 16:50 NZ104 (SYD to AKL)
+
** Cooper Lees - [https://twitter.com/cooperlees @cooperlees]
+
** Patrick Shuff
+
* 17:15 NZ124 (MEL to AKL)
+
** Mark Jessop (@darksidelemm - from VA206 departing ADL)
+
* 17:30 JQ213 (MEL to AKL)
+
** David Bell (@dtbell91)
+
** Matthew Cengia (@mattcen)
+
** Mike Abrahall (@mijofa1)
+
** Josh Mesilane (@zindello)
+
** David Rowe
+
* 17:30 VA7436 (BNE to AKL)
+
** Ryan Stuart (@rstuart85) - have a free entry pass to the Virgin Lounge for anyone else on this flight. Contact me for details.
+
** Sven Dowideit (@SvenDowideit) - I'm in an AirBnB apartment near the venue, and was planning on taking the hour long AirBus
+
* 17:50 VA7430 (SYD to AKL)
+
** Mark Walkom (@warkolm)
+
* 18:20 VA7452/NZ972 (ADL to AKL)
+
** [mailto:ubermonk@gmail.com Andrew Kirkpatrick]
+
** Thomas Sprinkmeier
+
* 19:25 JQ260 (WLG to AKL)
+
** Jim (James) Whittaker
+
* 20:30 NZ476 (WLG to AKL)
+
** Douglas Bagnall
+
* 20:50 NZ118 (SYD to AKL)
+
** [[User:Paul Warren|Paul Warren]]
+
* 22:05 QF147 (SYD to AKL)
+
** John Dalton ([https://twitter.com/johndalton @johndalton])
+
  
==== Monday 12 Jan ====
+
Red Hat's Beaker integration and testing system provides such a
 +
service. This talk will describe the inventory gathering component of
 +
Beaker including the transition from using ""smolt"" to ""lshw"" for this
 +
component and how the Beaker team is able to use access to more esoteric hardware to enhance the capabilities of lshw.
  
* 14:35 EK434 (BNE to AKL)
 
** [[User:Fraser Tweedale|Fraser Tweedale]]
 
  
 +
=== Testing the cloud in the cloud ===
  
===Departures===
+
by Steve Kowalik
  
Order the entries below by scheduled '''time of departure''' from Auckland. ''Please use 24 hour clock format (hh:mm)'' and ''Auckland time!''
+
OpenStack makes heavy use of CI, spinning up instances to run tests, and then destroying them. This gets much harder when what you're trying to test on the cloud is if your cloud can deploy a cloud. In this talk I'll talk about what we're currently doing with CI in TripleO (OpenStack on OpenStack, that is, using OpenStack to deploy OpenStack) and what our future plans with the CI is.
  
==== Friday 16 Jan ====
 
*19:55 JQ283 (AKL to WLG)
 
** Jim (James) Whittaker
 
* 20:45 CI54 (AKL to BNE)
 
** [[User:Fraser Tweedale|Fraser Tweedale]]
 
  
==== Saturday 17 Jan ====
+
=== The Best Test Data is Random Test Data  ===
* Morning sometime - Driving to Wellington
+
** [[User:Andrew Ruthven|Andrew Ruthven]], Susanne Ruthven and kids
+
* 06:55 QF152 (AKL to MEL)
+
** John Dalton ([https://twitter.com/johndalton @johndalton])
+
* 07:00 NZ101 (AKL to SYD)
+
** [mailto:himangi774@gmail.com Himangi Saraogi]
+
* 07:00 VA7401 (SYD to AKL)
+
** Mark Walkom (@warkolm)
+
* 07:35 JQ202 (AKL to SYD)
+
** Michael Cordover (aka mjec)
+
* 09:00 VA7403 (AKL to SYD)
+
** [[User:Tim Serong|Tim Serong]] ([https://twitter.com/tserong @tserong])
+
* 09:30 NZ135 (AKL to BNE)
+
** [https://twitter.com/bradleymarshall Brad Marshall]
+
* 10:00 NZ721 (AKL to MEL)
+
** Thomas Sprinkmeier
+
* 10:40 NZ4 (AKL to LAX)
+
** [http://alasdairallan.com Alasdair Allan]
+
* 10:50 QF112 (AKL to PER)
+
** [[User:Andrew Buckeridge|Andrew Buckeridge]]
+
* 11:00 QF182 (AKL to SYD)
+
** Mike Carden
+
* 12:00 NZ429 (AKL to WLG)
+
** [[User:James Forman|James Forman]]
+
* 13:00 VA7419 and NZ119 (AKL to SYD)
+
** [mailto:ubermonk@gmail.com Andrew Kirkpatrick]
+
** Mark Jessop (@darksidelemm - then onto ADL on VA436)
+
** [[User:Paul Warren|Paul Warren]]
+
* 13:00 NZ439 (AKL to WLG)
+
** Douglas Bagnall
+
** [[User:Ewen McNeill|Ewen McNeill]]
+
* 13:10 VA165 (AKL to OOL)
+
** [[User:Katie Miller|Katie Miller]]
+
* 14:00 QF0144 (AKL to SYD)
+
** [mailto:lca@eyal.emu.id.au Eyal Lebedinsky]
+
** [[User:Jonathan Woithe|Jonathan Woithe]] (then on QF743 to ADL)
+
** [mailto:diego@biurrun.de Diego Biurrun]
+
* 14:25 VA7445 (AKL to PER)
+
** Byron Jones (glob)
+
* 14:30 NZ4995 (AKL to HKG)
+
** Jussi Pakkanen
+
* 14:45 VA7439 (AKL to BNE)
+
** Ryan Stuart (@rstuart85) - have a free entry pass to the Virgin Lounge for anyone else on this flight. Contact me for details.
+
** Sven Dowideit (@SvenDowideit) - /me waves at Ryan and yells `twins!`
+
* 15:25 VA7425 (AKL to MEL)
+
** [https://twitter.com/smarthall Daniel Hall]
+
* 17:00 QF154 (AKL to MEL)
+
** [mailto:jon@oxer.com.au Jonathan Oxer]
+
* 18:10 QF8771/EK435 (AKL to BNE)
+
** [http://www.humbug.org.au/RussellStuart Russell Stuart]
+
** [[User:Mark Ellem|Mark Ellem]]
+
** Jared Ring
+
** Lee Symes
+
** Matt Franklin
+
* 18:30 EK413 (AKL to SYD)
+
** [[User:Jeremy Visser|Jeremy Visser]]
+
* 18:50 EK407 (AKL to MEL)
+
** Kathy Reid (@kathyreid) and Sue Reid
+
** [https://twitter.com/fukawi2 Phillip Smith]
+
* 20:30 JQ214 (AKL to MEL)
+
** David Bell (@dtbell91)
+
** Matthew Cengia (@mattcen)
+
** Mike Abrahall (@mijofa1)
+
** Josh Mesilane (@zindello)
+
* 22:45 NZ2 (AKL to LAX)
+
** [mailto:bdale@gag.com Bdale Garbee]
+
** [mailto:keithp@keithp.com Keith Packard]
+
  
==== Sunday 18 Jan ====
+
by Fraser Tweedale
* 07:00 VA153 (AKL to BNE)
+
** Anthony Towns
+
* 08:15 QF142 (AKL to SYD)
+
** Steve Walsh
+
* 08:20 NZ123 (AKL to MEL)
+
** [[User:Stewart_Smith|Stewart Smith]]
+
* 14:25 VA7445 (AKL to PER)
+
** James 'Ender' Brown
+
* 14:30 MH-130 (AKL to KUL)
+
** [[User:Raghavendra Prabhu|Raghavendra Prabhu]]
+
* 16:00 VA7434 (WLG to BNE)
+
** Clinton Roy
+
* 18:10 QF8771/EK435 (AKL to BNE)
+
** James Iseppi
+
* 18:30 QF8763 (AKL to SYD)
+
** [mailto:danbryan@gmail.com Daniel Bryan]
+
* 18:40 VA0729/NZ0729 (AKL to MEL)
+
** Bianca Gibson
+
  
==== Monday 19 Jan ====
+
Testing accounts for a large portion of the cost of software
* 14:25 NZ175 (AKL to PER)
+
development.  Tools to automate testing allow for more thorough
** [mailto:me@kye.id.au Kye Russell]
+
testing in less time. *Property-based testing* provides ways to
** Delan Azabani
+
define expected properties of functions under test, and mechanisms
** Brock York
+
to automatically check whether those properties hold in a large
** Josh Batchelor
+
number of cases - or whether a property can be falsified.
** Luke Mercuri
+
** Clayton Johnson
+
** Matthew Pen
+
* 16:20 QF146 (AKL to SYD)
+
** [mailto:kibelan@gmail.com Ben Kelly]
+
  
==== Tuesday 20 Jan ====
+
This talk will establish the motivations and explore the mechanisms
* 08:15 QF142 (AKL to SYD)
+
of property-based testing.  Concepts will be demonstrated primarily
** Hugh Blemings
+
using Haskell's *QuickCheck* library.  We will also review property-based testing solutions for other popular languages.
** Rachael Blemings
+
  
==== Wednesday 21 Jan ====
+
The talk will conclude with a discussion of the limitations of
 +
property-based testing, and alternative approaches.
  
* 14:00 QF144 (AKL to SYD)
+
=== Developers, sysadmins, and everyone else: Why you should be using Serverspec ===
** Jason Lewis
+
** [mailto:remi@remlab.net Rémi Denis-Courmont] (onto QF5 arriving SIN)
+
  
==== Sunday 25 Jan ====
+
by Sarah Kowalik & Jesse Reynolds
* 08:20 VA7453 (AKL to ADL)
+
 
** [https://twitter.com/gm_stack Geordie Millar]
+
"Congratulations! You’ve shipped a package to your users! Your responsibilities as a maintainer are now complete! " – nobody, ever.
* 09:30 VA7435 (AKL to BNE)
+
 
** Michael Wheeler
+
Anyone who has shipped software knows that a release is just the beginning of another wave of pain on the beach of software maintainership. Does the package actually install? Does the package contain what’s intended? Does your software run as expected? These are all questions that most projects can’t answer easily at release time without significant manual testing.
 +
 
 +
Wouldn’t it be great if you could simulate the steps users will go through to get your software installed, and verify you meet their expectations?
 +
 
 +
Enter Serverspec, a simple test harness for behaviour testing running systems. Serverspec helps you do outside-in testing of your software across multiple platforms, by providing simple shortcuts for common tests like “is this service running with these arguments?”, “does the application boot with this line in the configuration file?”, or “have I opened a back door to attackers by creating a world writeable log file?”.
 +
 
 +
In this talk we’ll explore how to write and run automated Serverspec tests for your packages at release time, how to maintain and improve quality across multiple target install platforms, and ways you can save time when testing for regressions, behaviour, and the user install and configuration experience.
 +
 
 +
 
 +
=== Subunit2SQL: Tracking Individual Test Results in OpenStack's CI System  ===
 +
 
 +
by Matthew Treinish
 +
 
 +
The OpenStack project's CI system operates at a scale that is too large to feasibly track all the test results or to monitor the system health at any given time by hand. To automate this there are several programmatic resources available to track the results from the system, including tools like logstash and graphite. Using and building off these tools have been invaluable especially as the OpenStack project continues to grow. However, the granularity of data available was still mostly limited to the individual test job, which limited the type of analysis which could be done.
 +
 
 +
For some time there was a desire to track the performance of individual tests over a longer period of time to provide data for changes to testing policy and also to watch for performance regressions. However, there was no mechanism available to aggregate all the necessary data, nor a programmatic interface to automate extracting useful trends from it. To fill this gap subunit2sql was created. Subunit2sql takes the subunit output from the test runs, parses it, and stores the results in a SQL database. Having the result data easily available in a database at the granularity of individual tests has enabled the OpenStack project to better track the project's development and quality over time.
 +
 
 +
This talk will cover a basic outline of the subunit2sql architecture and data model, how it's deployed and used in the OpenStack CI environment, and provide an overview of the results and advantages that the OpenStack project has seen from using a subunit2sql DB to track test results.
 +
 
 +
=== Corpus collapsum: Partition tolerance of Galera put to test ===
 +
 
 +
by Raghavendra Prabhu
 +
 
 +
All real world networks do tend to fail every now and then.  Failures are common, however, resilience of distributed systems to partitions is not nearly ubiquitous enough. Even though partition tolerance is an integral part of Brewer's CAP theorem, distributed systems, even the ones slated to fulfill the 'P' in CAP, fail to meet it to desired levels or deterministic enough. This, unfortunately, is not an exception, but has become more commonplace as the recent research shows [1].
 +
 
 +
Galera [2] adds synchronous replication to MySQL/PXC (Percona XtraDB Cluster)[3] through wsrep replication API. Synchronous replication, requires not just a quorum but a consensus. In a noisy environment, a delayed consensus can delay a commit, thus add significant latency or even partition the network into multiple non-primary components and single primary component (if plausible). Hence, building resilience is not an expectation but a fundamental requirement.
 +
 
 +
This talk is about testing partition immunity of Galera. Docker and Netem/tc (traffic control) are used prominently here. Netem is important to simulate real-world failure events - packet loss, delay, corruption, duplication, reordering et.al., and to model real-world failure distributions like pareto, paretonormal, uniform etc. Docker, or containers in general, are essential to simulate multiple nodes which can be built at runtime, brought up easily, tore down, their networks and flows altered elegantly when and then required, quick horizontal scaling; performance is also kept in mind when choosing containers over full virtualization and others.  Sysbench OLTP is used for load generation, though, RQG (random query generator) can also be used here for advanced fuzz testing.
 +
 
 +
Salient observations discussed will be:
 +
 
 +
* Application of WAN segment-aware loss coefficents to virtual network interfaces.
 +
* Varying reconciliation periods after network noise is withdrawn.
 +
* Multi-node loss and short-lived noise burst visa-vis single-node loss and longer noise envelope.
 +
* Full-duplex linking of containers with dnsmasq.
 +
* Effects of non-network actors like slow/fast disks on fsync.
 +
* Round-robin request distribution to nodes with/without the nodes with network failures in chain.
 +
* Pre and post-testing sanity tests.
 +
* Log collection and analysis.
 +
* Horizontal scaling of test nodes and issues with Docker/namespace.
 +
 
 +
To conclude,  all the ins-and-outs of partition tolerance testing with Docker and Netem for Galera will be discussed. Other similar tools/frameworks like jespen [4] will also be discussed. Comparison of Galera/EVS (extended virtual synchrony) to other consensus protocols like Paxos, Raft etc. will also be made. Results of testing - addition of auto_evict to Galera - will also be highlighted at the end.
 +
 
 +
[1] https://queue.acm.org/detail.cfm?id=2655736
 +
[2] http://galeracluster.com/products/
 +
[3] http://www.percona.com/software/percona-xtradb-cluster
 +
[4] https://github.com/aphyr/jepsen
 +
 
 +
=== Kickstart new developers using Docker ===
 +
 
 +
by Sven Dowideit
 +
 
 +
Docker containers allow new developers to make quick contributions to your project without needing to first learn how to set up an environment.
 +
 
 +
These new developers can be be sure that the environment they're using to test their changes are the same as those used by everyone else, so asking for help is going to be easy.
 +
 
 +
Next up, you can use the same Docker built environment to run tests.... continuously - and suddenly, you're building your releases the same totally repeatable way.
 +
 
 +
=== Large Scale Identification of Race Conditions (In OpenStack CI) ===
 +
 
 +
by Joe Gordon
 +
 
 +
Does your project have a CI system that suffers from an ever-growing set of race conditions? We have the tool for you: it has enabled increased velocity despite project growth.
 +
 
 +
When talking about the GNU HURD kernel, Richard Stallman once said, “it turned out that debugging these asynchronous multithreaded programs was really hard.” With 30+ asynchronous services developed by over 1000 people the OpenStack project is an object lesson of this problem. One of the consequences is race conditions often leak into code with no obvious defect. Just before OpenStack’s most recent stable release we were pushing the boundaries of what was possible with manual tracking of race conditions. To address this problem we have developed an ElasticSearch based toolchain called “elastic-recheck.” This helps us track race conditions so developers can fix them and identify when CI failures are related to the failed patch or are due to a known pre-existing race condition. Automated tracking of over 70 specific race conditions has allowed us to quickly determine which bugs are hurting us the most, allowing us to prioritize debugging efforts. Immediate and automated classification of test failures into genuine and false failures has saved countless hours that would have been wasted digging through the over 350MBs of logs produced by a single test run.
 +
 
 +
=== Gerrit & Gertty: A Daily Habit ===
 +
 
 +
by Anita Kuno
 +
 
 +
Taking the functionality of Gerrit and turning it into a workflow friendly enough to become a regular part of one's workday, is the focus of this talk. Many folks are familiar with Gerrit but don't have the pieces necessary to complete the high workflow output of some Gerrit reviewers. This presentation will share some aspects of using Gerrit and Gertty which high volume reviewers use with the intention of helping those who wish to increase their Gerrit throughput.
 +
 
 +
=== Incorporating the Security Development Life Cycle and Static Code Analysis into our Everyday Development Lives: An Overview of Theory and Techniques. ===
 +
 
 +
by Dr. Jason Cohen
 +
 
 +
It would seem that, despite the exponential growth in security products, security services, security companies, security certifications, and general interest in the security topic; we are still bombarded with a constant parade of security vulnerability disclosures on a seemingly daily basis. It turns out that we in the Open Source community can no longer shake a disapproving finger at the closed-source giants without also pointing to ourselves and asking what we can do better.  Issues like the OpenSSL Heartbleed vulnerability and several Drupal issues over the past year represent a case in point of major security flaws in common Open Source software.  It also demonstrated that, in this era of increasingly modular code development and reuse of common libraries, we need to begin considering the impact of potential flaws in code we assume to be secure due simply to its widespread use and Open Source nature.  So, what do we do?  Although it’s not a magical solution or panacea to the problem; implementing Security Development Life-cycle best practices and principles for each and every software development endeavor we undertake (whether it is for your job or for an Open Source Project) can go a long way to reducing the potential for common security flaws.  In addition, there is no reason that Static Code Analysis should not be part of every development effort.  We are still seeing obvious, easy to fix flaws in modern source code. Input sanitization issues, Cross-Site-Scripting, buffer overflows, and many other known issues still represent the bulk of security issues present. Static Code Analysis can help catch many of these unnoticed issues before code makes it out of the developer’s hands. In addition, we can perform our own analysis on libraries that we wish to leverage to help determine risk ourselves.  In this talk, we will explore some common best practice Security Development Life-cycle theory and how we can integrate this into modern code development schemes such as Continuous Integration and Agile.  We will also look at how to integrate Static Code analysis tools into the development process, to include a demo of HP Fortify with Eclipse and an example analysis of a common Open Source code base.

Latest revision as of 12:21, 6 January 2015

Developer, Testing, Release and Continuous Integration Automation Miniconf 2015

This miniconf is all about improving the way we produce, collaborate, test and release software.

We want to cover tools and techniques to improve the way we work together to produce higher quality software:

  • code review tools and techniques (e.g. gerrit)
  • continuous integration tools (e.g. jenkins)
  • CI techniques (e.g. gated trunk, zuul)
  • testing tools and techniques (e.g. subunit, fuzz testing tools)
  • release tools and techniques: daily builds, interacting with distributions, ensuring you test the software that you ship.
  • applying CI in your workplace/project


Schedule


Beaker's Hardware Inventory System

By Nick Coghlan

Ever wondered what it might take to track down and resolve a kernel bug that only affects one particular variant of one particular processor architecture from one particular vendor? Step one is going to be actually getting hold of a suitable machine, and for that, you'll need an inventory system that provides that level of detail.

Red Hat's Beaker integration and testing system provides such a service. This talk will describe the inventory gathering component of Beaker including the transition from using ""smolt"" to ""lshw"" for this component and how the Beaker team is able to use access to more esoteric hardware to enhance the capabilities of lshw.


Testing the cloud in the cloud

by Steve Kowalik

OpenStack makes heavy use of CI, spinning up instances to run tests, and then destroying them. This gets much harder when what you're trying to test on the cloud is if your cloud can deploy a cloud. In this talk I'll talk about what we're currently doing with CI in TripleO (OpenStack on OpenStack, that is, using OpenStack to deploy OpenStack) and what our future plans with the CI is.


The Best Test Data is Random Test Data

by Fraser Tweedale

Testing accounts for a large portion of the cost of software development. Tools to automate testing allow for more thorough testing in less time. *Property-based testing* provides ways to define expected properties of functions under test, and mechanisms to automatically check whether those properties hold in a large number of cases - or whether a property can be falsified.

This talk will establish the motivations and explore the mechanisms of property-based testing. Concepts will be demonstrated primarily using Haskell's *QuickCheck* library. We will also review property-based testing solutions for other popular languages.

The talk will conclude with a discussion of the limitations of property-based testing, and alternative approaches.

Developers, sysadmins, and everyone else: Why you should be using Serverspec

by Sarah Kowalik & Jesse Reynolds

"Congratulations! You’ve shipped a package to your users! Your responsibilities as a maintainer are now complete! " – nobody, ever.

Anyone who has shipped software knows that a release is just the beginning of another wave of pain on the beach of software maintainership. Does the package actually install? Does the package contain what’s intended? Does your software run as expected? These are all questions that most projects can’t answer easily at release time without significant manual testing.

Wouldn’t it be great if you could simulate the steps users will go through to get your software installed, and verify you meet their expectations?

Enter Serverspec, a simple test harness for behaviour testing running systems. Serverspec helps you do outside-in testing of your software across multiple platforms, by providing simple shortcuts for common tests like “is this service running with these arguments?”, “does the application boot with this line in the configuration file?”, or “have I opened a back door to attackers by creating a world writeable log file?”.

In this talk we’ll explore how to write and run automated Serverspec tests for your packages at release time, how to maintain and improve quality across multiple target install platforms, and ways you can save time when testing for regressions, behaviour, and the user install and configuration experience.


Subunit2SQL: Tracking Individual Test Results in OpenStack's CI System

by Matthew Treinish

The OpenStack project's CI system operates at a scale that is too large to feasibly track all the test results or to monitor the system health at any given time by hand. To automate this there are several programmatic resources available to track the results from the system, including tools like logstash and graphite. Using and building off these tools have been invaluable especially as the OpenStack project continues to grow. However, the granularity of data available was still mostly limited to the individual test job, which limited the type of analysis which could be done.

For some time there was a desire to track the performance of individual tests over a longer period of time to provide data for changes to testing policy and also to watch for performance regressions. However, there was no mechanism available to aggregate all the necessary data, nor a programmatic interface to automate extracting useful trends from it. To fill this gap subunit2sql was created. Subunit2sql takes the subunit output from the test runs, parses it, and stores the results in a SQL database. Having the result data easily available in a database at the granularity of individual tests has enabled the OpenStack project to better track the project's development and quality over time.

This talk will cover a basic outline of the subunit2sql architecture and data model, how it's deployed and used in the OpenStack CI environment, and provide an overview of the results and advantages that the OpenStack project has seen from using a subunit2sql DB to track test results.

Corpus collapsum: Partition tolerance of Galera put to test

by Raghavendra Prabhu

All real world networks do tend to fail every now and then. Failures are common, however, resilience of distributed systems to partitions is not nearly ubiquitous enough. Even though partition tolerance is an integral part of Brewer's CAP theorem, distributed systems, even the ones slated to fulfill the 'P' in CAP, fail to meet it to desired levels or deterministic enough. This, unfortunately, is not an exception, but has become more commonplace as the recent research shows [1].

Galera [2] adds synchronous replication to MySQL/PXC (Percona XtraDB Cluster)[3] through wsrep replication API. Synchronous replication, requires not just a quorum but a consensus. In a noisy environment, a delayed consensus can delay a commit, thus add significant latency or even partition the network into multiple non-primary components and single primary component (if plausible). Hence, building resilience is not an expectation but a fundamental requirement.

This talk is about testing partition immunity of Galera. Docker and Netem/tc (traffic control) are used prominently here. Netem is important to simulate real-world failure events - packet loss, delay, corruption, duplication, reordering et.al., and to model real-world failure distributions like pareto, paretonormal, uniform etc. Docker, or containers in general, are essential to simulate multiple nodes which can be built at runtime, brought up easily, tore down, their networks and flows altered elegantly when and then required, quick horizontal scaling; performance is also kept in mind when choosing containers over full virtualization and others. Sysbench OLTP is used for load generation, though, RQG (random query generator) can also be used here for advanced fuzz testing.

Salient observations discussed will be:

  • Application of WAN segment-aware loss coefficents to virtual network interfaces.
  • Varying reconciliation periods after network noise is withdrawn.
  • Multi-node loss and short-lived noise burst visa-vis single-node loss and longer noise envelope.
  • Full-duplex linking of containers with dnsmasq.
  • Effects of non-network actors like slow/fast disks on fsync.
  • Round-robin request distribution to nodes with/without the nodes with network failures in chain.
  • Pre and post-testing sanity tests.
  • Log collection and analysis.
  • Horizontal scaling of test nodes and issues with Docker/namespace.

To conclude, all the ins-and-outs of partition tolerance testing with Docker and Netem for Galera will be discussed. Other similar tools/frameworks like jespen [4] will also be discussed. Comparison of Galera/EVS (extended virtual synchrony) to other consensus protocols like Paxos, Raft etc. will also be made. Results of testing - addition of auto_evict to Galera - will also be highlighted at the end.

[1] https://queue.acm.org/detail.cfm?id=2655736 [2] http://galeracluster.com/products/ [3] http://www.percona.com/software/percona-xtradb-cluster [4] https://github.com/aphyr/jepsen

Kickstart new developers using Docker

by Sven Dowideit

Docker containers allow new developers to make quick contributions to your project without needing to first learn how to set up an environment.

These new developers can be be sure that the environment they're using to test their changes are the same as those used by everyone else, so asking for help is going to be easy.

Next up, you can use the same Docker built environment to run tests.... continuously - and suddenly, you're building your releases the same totally repeatable way.

Large Scale Identification of Race Conditions (In OpenStack CI)

by Joe Gordon

Does your project have a CI system that suffers from an ever-growing set of race conditions? We have the tool for you: it has enabled increased velocity despite project growth.

When talking about the GNU HURD kernel, Richard Stallman once said, “it turned out that debugging these asynchronous multithreaded programs was really hard.” With 30+ asynchronous services developed by over 1000 people the OpenStack project is an object lesson of this problem. One of the consequences is race conditions often leak into code with no obvious defect. Just before OpenStack’s most recent stable release we were pushing the boundaries of what was possible with manual tracking of race conditions. To address this problem we have developed an ElasticSearch based toolchain called “elastic-recheck.” This helps us track race conditions so developers can fix them and identify when CI failures are related to the failed patch or are due to a known pre-existing race condition. Automated tracking of over 70 specific race conditions has allowed us to quickly determine which bugs are hurting us the most, allowing us to prioritize debugging efforts. Immediate and automated classification of test failures into genuine and false failures has saved countless hours that would have been wasted digging through the over 350MBs of logs produced by a single test run.

Gerrit & Gertty: A Daily Habit

by Anita Kuno

Taking the functionality of Gerrit and turning it into a workflow friendly enough to become a regular part of one's workday, is the focus of this talk. Many folks are familiar with Gerrit but don't have the pieces necessary to complete the high workflow output of some Gerrit reviewers. This presentation will share some aspects of using Gerrit and Gertty which high volume reviewers use with the intention of helping those who wish to increase their Gerrit throughput.

Incorporating the Security Development Life Cycle and Static Code Analysis into our Everyday Development Lives: An Overview of Theory and Techniques.

by Dr. Jason Cohen

It would seem that, despite the exponential growth in security products, security services, security companies, security certifications, and general interest in the security topic; we are still bombarded with a constant parade of security vulnerability disclosures on a seemingly daily basis. It turns out that we in the Open Source community can no longer shake a disapproving finger at the closed-source giants without also pointing to ourselves and asking what we can do better. Issues like the OpenSSL Heartbleed vulnerability and several Drupal issues over the past year represent a case in point of major security flaws in common Open Source software. It also demonstrated that, in this era of increasingly modular code development and reuse of common libraries, we need to begin considering the impact of potential flaws in code we assume to be secure due simply to its widespread use and Open Source nature. So, what do we do? Although it’s not a magical solution or panacea to the problem; implementing Security Development Life-cycle best practices and principles for each and every software development endeavor we undertake (whether it is for your job or for an Open Source Project) can go a long way to reducing the potential for common security flaws. In addition, there is no reason that Static Code Analysis should not be part of every development effort. We are still seeing obvious, easy to fix flaws in modern source code. Input sanitization issues, Cross-Site-Scripting, buffer overflows, and many other known issues still represent the bulk of security issues present. Static Code Analysis can help catch many of these unnoticed issues before code makes it out of the developer’s hands. In addition, we can perform our own analysis on libraries that we wish to leverage to help determine risk ourselves. In this talk, we will explore some common best practice Security Development Life-cycle theory and how we can integrate this into modern code development schemes such as Continuous Integration and Agile. We will also look at how to integrate Static Code analysis tools into the development process, to include a demo of HP Fortify with Eclipse and an example analysis of a common Open Source code base.