Difference between revisions of "OpenZFS Developer Summit"

From OpenZFS
Jump to: navigation, search
(Topics that were discussed)
Line 2: Line 2:
 
* to foster cross-platform community discussions of OpenZFS work
 
* to foster cross-platform community discussions of OpenZFS work
 
* to make progress on some of the projects proposed for this community.
 
* to make progress on some of the projects proposed for this community.
 
== Event Report ==
 
  
 
[https://www.youtube.com/playlist?list=PLaUVvul17xSdWMBt5tAC8Hu7bbeWskD_q Video recordings of the presentations] are in the OpenZFS channel on YouTube.
 
[https://www.youtube.com/playlist?list=PLaUVvul17xSdWMBt5tAC8Hu7bbeWskD_q Video recordings of the presentations] are in the OpenZFS channel on YouTube.
Line 9: Line 7:
 
[https://www.surveymonkey.com/s/SL98Z8H Participant Survey]: Please complete by December 6 end of day.
 
[https://www.surveymonkey.com/s/SL98Z8H Participant Survey]: Please complete by December 6 end of day.
  
=== Topics that were discussed ===
+
== Topics that were discussed ==
 
{| class="wikitable"
 
{| class="wikitable"
 
!Title!!Speaker!!Slides!!Video
 
!Title!!Speaker!!Slides!!Video
Line 42: Line 40:
 
|}
 
|}
  
=== Hackathon projects ===
+
== Hackathon projects ==
 
These projects were worked on at the hackathon:
 
These projects were worked on at the hackathon:
  
Line 55: Line 53:
 
* onexit for osx
 
* onexit for osx
  
== A huge THANK YOU to our generous sponsors ==
+
== Event Planning Info ==
{|cellpadding="10"
+
See [[OpenZFS Developer Summit 2013 planning]]
|-
+
|[[File: Delphix logo.png|200x200px|link=http://delphix.com]]
+
|[[File: Ddrdrive logo dev 2013.png|200x200px|link=http://www.ddrdrive.com]]
+
|[[File: Logo freebsdfoundation.png|200x200px|link=http://www.freebsd.org]]
+
|-
+
|[[FIle: SageCloud-logo-RGB-large-optimized.jpg|200x200px|link=http://sagecloud.com/]]
+
|[[File: Ixsystems.jpg|200x200px|link=http://www.ixsystems.com/]]
+
|[[File: CS logo withTagline 150px.png|200x200px|link=http://cloudscaling.com/]]
+
|-
+
|[[File: SpectraLogicLogo.jpg|200x200px|link=http://www.spectralogic.com/]]
+
|[[File: Joyent-logo.png|200x200px|link=http://joyent.com]]
+
|[[File: New_hac_logo.jpg|200x200px|link=http://www.high-availability.com/]]
+
|-
+
|[[File: Nexenta logo.jpg|200x200px|link=http://www.nexenta.com/]]
+
|}
+
 
+
= Event Planning Info =
+
 
+
== Schedule ==
+
=== Sunday, November 17th ===
+
{|border="1" cellpadding="10"
+
|-
+
|6:30pm
+
| Optional dinner at [http://goo.gl/943i5i Rosamunde Sausage Grill] on Mission St.
+
Location is a 3 minute walk from the 24th Street BART station.
+
|}
+
 
+
=== Monday, November 18th ===
+
{|border="1" cellpadding="10"
+
|-
+
|valign="top"|8:30am - 9:00am
+
| Registration and breakfast
+
|-
+
|valign="top"|9:00am - 9:15am
+
| Introductions
+
|-
+
|valign="top"|9:15am - 9:45am
+
| State of OpenZFS & Event overview (Matt Ahrens)
+
|-
+
|valign="top"|9:45am - 10:00am
+
| List and prioritize discussion topics (lead by Matt Ahrens & Karyn Ritter)
+
|-
+
|valign="top"|10:00am - 11:00am
+
| Discussion sessions
+
|-
+
|valign="top"|11:00am - 11:15am
+
| Coffee & snack break
+
|-
+
|valign="top"|11:15am - 12:15pm
+
| Discussion sessions
+
|-
+
|valign="top"|12:15am - 1:00pm
+
| Lunch
+
|-
+
|valign="top"|1:00pm - 1:30pm
+
| Discussion session: Community Management (lead by Karyn Ritter)
+
|-
+
|valign="top"|1:30pm - 3:00pm
+
| Discussion sessions
+
|-
+
|valign="top"|3:00pm - 3:30pm
+
| Coffee & snack break
+
|-
+
|valign="top"|3:30pm - 5:30pm
+
| Discussion sessions
+
|-
+
|valign="top"|5:30pm - 5:45pm
+
| Closing (Matt Ahrens)
+
|-
+
|valign="top"|5:45pm
+
| Beer bash & casual dinner at Delphix office
+
|}
+
 
+
=== Tuesday, November 19th ===
+
{|border="1" cellpadding="10"
+
|-
+
|valign="top"|9:00am - 9:30am
+
| Breakfast
+
|-
+
|valign="top"|9:30am - 10:00am
+
| Discuss Hackathon ideas & form teams
+
|-
+
|valign="top"|10:00am - 12:30pm
+
|Hackathon
+
|-
+
|valign="top"|12:30pm - 1:30pm
+
|Lunch
+
|-
+
|valign="top"|1:30pm - 5:00pm
+
|Hackathon
+
|-
+
|valign="top"|5:00pm - 5:45pm
+
| Hackathon presentations
+
|-
+
|valign="top"|6:00pm
+
| Optional dinner at [https://www.google.com/maps/preview#!q=Thirsty+Bear+Brewery%2C+Howard+St%2C+San+Francisco%2C+CA&data=!1m4!1m3!1d3309!2d-122.4011122!3d37.7868869!4m15!2m14!1m13!1s0x8085807d0bcfd0df%3A0xdfac6306f9e18402!3m8!1m3!1d13236!2d-122.3926051!3d37.7913156!3m2!1i1261!2i806!4f13.1!4m2!3d37.785502!4d-122.399664 Thirsty Bear]
+
Location is 5 minute walk from Delphix office.
+
|}
+
 
+
Note that the optional, off-site dinners on Sunday and Tuesday are at your own expense.
+
 
+
== Location ==
+
The [http://www.delphix.com/company/about-delphix/ Delphix office] in San Francisco:
+
 
+
[https://www.google.com/maps/preview#!q=604+Mission+St.+Suite+400%2C+San+Francisco%2C+CA+94105&data=!1m4!1m3!1d26473!2d-122.4224157!3d37.7867658!4m12!2m11!1m10!1s0x80858062b5adcc23%3A0xd7584093492239e6!3m8!1m3!1d12612!2d-122.3926051!3d37.7913156!3m2!1i1024!2i768!4f13.1 604 Mission Street, 4th floor, San Francisco, CA 94105]
+
 
+
Use the keypad to call Delphix and you will be buzzed in.  Google maps has pretty good [http://goo.gl/QyUHUj directions].  The easiest way to reach the Delphix SF office is by public transportation.  Transportation options, easiest listed first:
+
* BART to the Montgomery St. Station + 3 minute walk
+
* Caltrain to the 4th & King station + 25 minutes walk
+
* Caltrain to the 4th & King station + bus (routes 10, 30, or 45)
+
* drive + park in a lot (costs about $25/day e.g. 41 Tehama Street, 509 Howard Street, or 55 Hawthorne Street)
+
 
+
== Discussion topics ==
+
'''Note: If you are in the room, you are potentially a speaker.''' This will be done in unconference style, so every attendee should come prepared to present or otherwise lead a discussion or session of up to 45 minutes.  The most important things to discuss in this setting are:
+
 
+
# Contentious topics that we need to get widespread agreement on
+
# Reports of work in progress that need input from others
+
 
+
==== Proposed topics ====
+
* OpenZFS code repository (Matt)
+
** what code changes are needed to make this a success
+
** process & procedure (code review, testing, committers)
+
* Performance on full or fragmented pools (George, Ned, Alfred)
+
** block allocator improvements
+
* Portable implementation of FMA functionality for Linux & FreeBSD (e.g. for hot spare activation) (Brian)
+
* OpenZFS community organization (Karyn, Jordan, Matt)
+
* Storage tiering (Boris)
+
* How to investigate ZFS performance problems (Adam)
+
* Test suite (John K)
+
* ZFS Channel Programs (Chris, Max G, Matt)
+
* how to examine the ZFS on-disk format with MDB (Max B)
+
* Interactions with virtual memory subsystem (Brian, Jorgen)
+
** dealing with swap on zfs
+
** how the VM system handles large memory allocations
+
* Scalability on large systems, known bottlenecks (Kirill)
+
* How to educate new ZFS developers (Max G)
+
* Disk management - identification, drive spin down/up (John L)
+
* Resumable send/recv (Boris, Max G, Chris)
+
 
+
====Topics that were discussed====
+
* Platform Panel (w/ representatives from illumos, FreeBSD, Linux, and OSX)
+
* Platform-independent code repository (Matt Ahrens)
+
* Storage tiering (Boris Protopopov)
+
* Vendor Panel (all represented companies sharing their work)
+
* Community planning (Karyn Ritter)
+
* Channel programs (Chris Siden & Max Grossman)
+
* Test coverage (John Kennedy)
+
* Performance investigations (Adam Leventhal)
+
* Performance on full/fragmented pools (George Wilson)
+
* Scalability (Kirill Davydychev)
+
* Virtual memory interactions (Brian Behlendorf)
+
* Multi tenancy (Rob Mustacchi)
+
* Examining on-disk format (Max Bruning)
+
 
+
== Hackathon ==
+
The goals of the hackathon are:
+
* to get people working with ZFS engineers from outside their normal circles
+
* to start work on some of the [[projects]] we'd like to accomplish
+
Therefore, we should plan to work in small teams of 2-3 engineers.  The team members could be pair-programming, splitting up the work into discrete tasks for each person, or advising / implementing.
+
 
+
=== Hackathon projects ===
+
These projects were worked on at the hackathon:
+
 
+
* Testrunner suite running on other platforms: linux, freebsd (+ illumos libzpool)
+
* channel programs: pass instruction stream into kernel
+
* ZFS send progress reporting with "set proc title"
+
* dedicated kmem_cache for dedup table entries
+
* larger (1MB) blocksize
+
* compressed ARC
+
* upstreaming dbuf_user & dbuf state machine (spectra DMU changes)
+
* limit DDT size
+
* onexit for osx
+
 
+
=== Hackathon ideas ===
+
''Add your idea and your name below.  Be prepared to explain your idea at the beginning of the hackathon and enlist/entice others to help with it.''
+
* Userland ioctls: support for send/recv (fd operations over the wire)  [[User:mahrens]]
+
* larger (1MB) blocksize: shouldn't be too hard if we ignore performance impact [[User:mahrens]]
+
* dedicated kmem_cache for dedup table entries
+
* stdin / stdout buffer to improve performance of zfs send and receive
+
* ZFS mechanism to override devices' reported sector size (512 vs 4k) (alex @ delphix)
+
* feature flags enhancements for send streams [[User:csiden]]
+
* set owner of all files within a filesystem with a ZFS property [[User:csiden]]
+
* dynamic taskqs for ZIO [[User:ahl]]
+
* towards Oracle Solaris ZFS compatibility [[User:ahl]]
+
* RAIDzN -- full, arbitrary RAID-7 [[User:ahl]]
+
* Testrunner suite running on other platforms (John Kennedy)
+
** or at least identify what is preventing it from running
+
 
+
== Registration ==
+
We are expecting everyone will be there to contribute/participate.
+
 
+
{|border="1" cellpadding="10"
+
|-
+
!| Name
+
!| Affiliation
+
!| What do you want to talk about?
+
!| Your experience with ZFS
+
!| Travel information
+
!| T-shirt Size
+
!| Food allergies/preferences? (please list)
+
|-
+
| [[User:mahrens | Matt Ahrens]]
+
| Delphix
+
| OpenZFS repo: what interfaces should be supported
+
| Founded ZFS, OpenZFS
+
| Local to SF
+
| M
+
| none
+
|-
+
|[[User:sanjeev | Sanjeev Bagewadi]]
+
| Pluribus Networks
+
|
+
| Was part of the RPE team in Sun which fixed ZFS issues faced by customers
+
| Will be in the Bay Area around that time
+
| M
+
| none
+
|-
+
|-
+
| Ned Bass
+
| LLNL
+
| Common code base, block allocator improvements
+
| ZFS on Linux port, Lustre integration
+
| from East Bay
+
| XL
+
| vegetarian preferred
+
|-
+
| [[User:behlendorf | Brian Behlendorf]]
+
| LLNL
+
| Portable memory management & FMA infrastructure
+
| ZFS on Linux Maintainer
+
| Local to east Bay Area
+
| XL
+
| none
+
|-
+
| Max Bruning
+
| Joyent
+
| examining ZFS on-disk format with MDB
+
| leads ZFS training classes
+
| ?
+
| ?
+
| ?
+
|-
+
| Steven Burgess
+
| Datto Inc.
+
| openZFS as a customer
+
| Working as a developer at a company whose product uses ZOL
+
| from NYC
+
| M
+
| none
+
|-
+
| [[User:kdavyd | Kirill Davydychev]]
+
| Nexenta
+
| Scalability on large systems, major known bottlenecks.
+
| Debugging, performance analysis/tuning
+
| Houston
+
| M
+
| none
+
|-
+
| Justin Gibbs
+
| Spectra Logic
+
| Ref-Links, Resilvering Algorithm, Spares, auto-configuration
+
| DMU performance work
+
| flying in from CO
+
| L
+
| vegetarian
+
|-
+
| [[User:Max | Max Grossman]]
+
| Delphix
+
| Making it easier to get new ZFS developers up to speed (retrospective on the hole birth feature)
+
| 3 months so far, still so much to learn...
+
| Local
+
| L
+
| none
+
|-
+
| Jordan Hubbard
+
| iXsystems
+
| OpenZFS community organization, independent code repo processes
+
| ?
+
| Local
+
| ?
+
| ?
+
|-
+
| Jeremy Jones
+
| Delphix
+
| ?
+
| ZFS bookmarks; debugging techniques
+
| local
+
| ?
+
| ?
+
|-
+
| [[User:jkennedy | John Wren Kennedy]]
+
| Delphix
+
| Uniform test coverage for OpenZFS
+
| Supported ZFS in RPE at Sun/Oracle, and test ZFS in QA at Delphix.
+
| Denver
+
| L
+
| none
+
|-
+
| [[User:jml | John Layman]]
+
| SageCloud
+
| Interested in disk management (spin up/down disks, drive identification, etc.)
+
| Been working with ZFS on Linux for a few months, contributed small bugfix
+
| Boston
+
| XL
+
| none
+
|-
+
| [[User:ahl | Adam Leventhal]]
+
| Delphix
+
| ZFS performance investigations?
+
| RAID-Z2/3, performance work, some small ZFS tasks here and there
+
| local
+
| L
+
| none
+
|-
+
| [[User:delphij | Xin Li]]
+
| iXsystems
+
| ZFS developer for FreeBSD, TrueNAS and FreeNAS products
+
| FreeBSD deputy security officer / release engineering team member
+
| local Bay Area
+
| L
+
| none
+
|-
+
| [[User:lundman | Jorgen Lundman]]
+
| GMO Internet
+
| ZFS, portability
+
| ZFS on OS X
+
| Tokyo
+
| XL
+
| none
+
|-
+
| [[User:maxim | Maxim Martynov]]
+
|
+
|
+
|
+
|
+
| M
+
| none
+
|-
+
| Robert Mustacchi
+
| Joyent
+
|
+
| Misc. debugging
+
| Local to SF
+
| L
+
| None
+
|-
+
| [[User:tonyn | Tony Nguyen]]
+
| Nexenta
+
|
+
| Misc. debugging, import and I/O performance
+
| flying in from Denver
+
| M
+
| None
+
|-
+
| Alfred Perlstein
+
| iXsystems
+
| ZFS performance and debugging
+
| FreeNAS (FreeBSD+ZFS) developer
+
| ?
+
| ?
+
| ?
+
|-
+
| Boris Protopopov
+
| Nexenta
+
| Resumable send/recv, defrag, storage tiering, spare management, doing fancy things with vdevs
+
| ZFS dev lead at Nexenta, ZFS extensions for enterprise class features and performance
+
| Boston
+
| M
+
| None
+
|-
+
| [[User:awreece | Alex Reece]]
+
| Delphix
+
|
+
| small bug fixes
+
| local
+
| M
+
| No marinera
+
|-
+
| [[User:kritter | Karyn Ritter]]
+
| Delphix
+
| Community management (discussion)
+
|
+
| local
+
| M
+
|
+
|-
+
| [[User:rodrigc | Craig Rodrigues]]
+
| iXsystems
+
| ZFS developer for TrueNAS and FreeNAS products
+
| FreeBSD release engineering team member
+
| local
+
| L
+
| none
+
|-
+
| [[User:csiden | Chris Siden]]
+
| Delphix
+
| ZFS Channel Programs, feature flags enhancements for send streams
+
| Worked on feature flags and async destroy
+
| local
+
| L
+
| none
+
|-
+
| Alex Solomatnikov
+
|
+
|
+
|
+
|
+
| M
+
| none
+
|-
+
| [[User:pks | Prashanth Sreenivasa ]]
+
|
+
| I would like to learn more about ZIL, ZFS I/O scheduler & pipeline and impact of using flash-disks instead of conventional hard-disks.
+
| Worked on various PDE (product development engineering) groups including Network Storage, Systems & Solaris kernel. Recently I'm working on ZFS based storage appliances, trying to improve zfs performance on illumos/x86 platform and looking forward to contribute to the open source zfs.
+
|
+
| M
+
|
+
|-
+
| [[User:prakashsurya | Prakash Surya]]
+
| LLNL
+
|
+
| Misc. work on Linux port. Much to learn, still.
+
| Local to east Bay Area
+
| M
+
| none
+
<!--
+
|-
+
| Ilya Usvyatsky
+
| Nexenta
+
| resumable send/recv, defrag
+
| ?
+
| ?
+
| ?
+
| ?
+
-->
+
|-
+
| Rafael Vanoni
+
| Pluribus
+
| I'm not a ZFS developer, but would like to work my way towards that.
+
| I worked on the thread scheduler/dispatcher at Sun/Oracle
+
| ?
+
| M
+
| ?
+
|-
+
| [[User:gwilson | George Wilson]]
+
| Delphix
+
| performance on full & fragmented pools
+
| Implemented lots of SPA & VDEV stuff
+
| flying in from Atlanta ~Monday - Wednesday
+
| L
+
| none
+
|-
+
| Matt Youds
+
| High Availability
+
|
+
| Experience with ZFS configuration at High-Availability.
+
|
+
| L
+
| Fish
+
|-
+
| Sam Zaydel
+
| ?
+
| ?
+
| ?
+
| ?
+
| ?
+
| ?
+
|}
+
 
+
== Sponsors ==
+
The event was fully sponsored!  Thank you to our sponsors:
+
 
+
* Breakfast (2 days) - Sponsored!  Thank you [http://www.cloudscaling.com Cloudscaling] and [http://www.ixsystems.com iXsystems]
+
* Lunch  (2 days) - Sponsored!  Thank you [http://sagecloud.com Sage Cloud], [http://www.spectralogic.com Spectra Logic] and [http://www.freebsdfoundation.org FreeBSD Foundation]
+
* Coffee Breaks (2 days) - Sponsored!  Thank you [http://www.ixsystems.com iXsystems]
+
* Beer Bash (day 1) - Sponsored!  Thank you [http://www.ddrdrive.com DDRdrive]
+
* T-Shirts  - Sponsored!  Thank you [http://www.dexlphix.com Delphix]
+
* Video Recordings - Thank you [http://www.joyent.com Joyent]
+
* Cash Donation for expenses - Thank you [http://www.nexenta.com Nexenta] and [http://www.high-availability.com/ High-Availability]
+
 
+
What do you get for sponsoring:
+
 
+
* Your name/link on this page!
+
* Thank you mention during the opening and closing
+
* Space to put your pop-up banner at the conference venue
+
* An opportunity to be known as a ZFS supporter in a very talented and influential community.
+

Revision as of 18:18, 22 November 2013

The OpenZFS Developer Summit was held November 18-19, 2013. The goals of the event were:

  • to foster cross-platform community discussions of OpenZFS work
  • to make progress on some of the projects proposed for this community.

Video recordings of the presentations are in the OpenZFS channel on YouTube.

Participant Survey: Please complete by December 6 end of day.

Topics that were discussed

Title Speaker Slides Video
Introduction Matt Ahrens slides video
Platform Panel Brian Behlendorf, Jorgen Lundman, Chris Siden, Xin Li - video
Platform-independent code repository Matt Ahrens slides video
Storage Tiering Boris Protopopov slides video
Vendor Lightning Talks Many - video
Community Planning Karyn Ritter slides video
Channel Programs Chris Siden, Max Grossman slides video
Test Coverage John Kennedy slides video
Performance Investigation Adam Leventhal slides video
Performance on full & fragmented pools George Wilson - video
Scalability Kirill Davydychev - video
Virtual Memory Interactions Brian Behlendorf - video
Multi Tenancy Rob Mustacchi - video
Examining On-disk Format Max Bruning - video

Hackathon projects

These projects were worked on at the hackathon:

  • Testrunner suite running on other platforms: linux, freebsd (+ illumos libzpool)
  • channel programs: pass instruction stream into kernel
  • ZFS send progress reporting with "set proc title"
  • dedicated kmem_cache for dedup table entries
  • larger (1MB) blocksize
  • compressed ARC
  • upstreaming dbuf_user & dbuf state machine (spectra DMU changes)
  • limit DDT size
  • onexit for osx

Event Planning Info

See OpenZFS Developer Summit 2013 planning