Eventbrite Engineering at PyConES

PyConES is the main event for Python developers in Spain and a must-attend for the engineering team based in Madrid. We meet the Python community every year, so we have a chance to catch up with fellow developers from other parts of the country. We learn a lot from them as we share our most recent experiences, either through our sessions or while hanging out with a coffee in hand.

The Spanish Python community has an identity of its own: it’s very diverse, open and most of all, inclusive. These values are essential to us, that’s why it’s not only necessary to be present at this great event, but also to sponsor it!

The trip to Málaga

PyConES is celebrated at a different location every year so that everybody gets to attend at least once, to share the hard work among various members of the community, and –why not— to showcase beautiful Spanish cities.

This year we attended PyConES at Málaga. This city is 530 km south of Madrid, and because that’s too far for many people, Eventbrite facilitated a bus so everybody could get there. That allowed us to meet some attendees before the event.

The Python Conference

Eventbrite, similar to PyConES, has a firm commitment to diversity, so we didn’t want to miss the opportunity to join the Django Girls workshop. (Once at the conference, we were glad to see that 25% of the attendees and 33% of the speakers were female!) One of our Britelings, Federico Mon (@gnufede), was a mentor for the workshop and he enjoyed it. So much so that we’re going to repeat the experience in Madrid on November 17th.

Our engineering team presented two talks:

  • Andrew Godwin(@andrewgodwin), who works as SRE in San Francisco, and is also a Django core developer, talked about the approach to make Django asynchronous.
  • Federico Mon (@gnufede), an engineering manager in Madrid, told us why Ticketea used Wagtail, and how can it be employed not only in blogs or news sites.

While this was the first time for Eventbrite to attend the conference, the Spanish team in Madrid had participated many times in the past years. Our faces were very recognizable, so obvious questions arose: “Where is Ticketea?”, “What’s Eventbrite?”. We were committed to satisfying the curiosity of everyone visiting our booth —which was a lot, by the way. Eventually, people got to know the brand and gained interest in it not as a platform, but also as a nice place to work in the center of Madrid. We met lots of people who want to come and visit us at our office!

When people visited our booth we had the chance to chit-chat with many of them, give them some Eventbrite goodies (very cool ones if you allow me), and discover their interests. We met many young people, a more significant amount of female attendees than at any other conference we attended this year. Among them were some non-devs that wanted to start with programming, and many data scientists as well. We can assert that the Python community is – hands down – one of the healthiest ones out there.

Also, we are hiring!

First of all, we want to thank the organizers of Python Spain, speakers, sponsors, and attendees for making it a great conference every year.

We are looking for passionate React and Django developers and also Site Reliability Engineers. We are expecting to welcome many more roles in the coming future, so if you are interested in working with us, don’t miss out and check our open positions at eventbrite.com/careers.

For more about other conferences Eventbrite has been attending so far, check this out!

Getting the most out of React Alicante

14 Britelings arrived by way of planes, trains, and automobiles to the city of Alicante. We joined close to 400 attendees at a two-day conference, React Alicante.

Keep reading to learn about the new stuff coming out in the React ecosystem and learn more about our favorite talks at the event.

The atmosphere

Set in the Melia hotel, overlooking the Mediterranean coast and its rocky shores, the conference venue location was enviable. Inside, a small group of sponsor tables lined the conference room lobby. Eventbrite’s table, stocked by our R+D Villena team, was the first one in the attendees’ line of sight as they made their way in to see the speakers. Our swag didn’t last long, but we made sure to keep the Eventbrite presence strong. We had 14 Britelings in the house and our very own Ben Ilegbodu speaking at the event.

The Lonely and Dark Road to Styling in React

Sara Vieira (@NikkitaFTW) walks us through the dark alleys of styling with CSS in ReactJS apps. Sara starts her talk by reassuring us that CSS is hard. We know all about that here at Eventbrite, where we use a design system to speed up our development process. This often keeps us from having to walk The Lonely and Dark Road to Styling in React. Still, Sara’s talk gives us a lot to think about when it comes to styling in ReactJS apps. She walks us through the pros and cons of everything from link tags to BEM, and her main focus, styled-components.

The Year of Web Components

Dominik Kundel(@DKundel) reminded us, once again, that web components are out there and this is their moment! What are web components and why should you care? Web components are a set of building blocks, defined as WC3 web standards, that allow the browser to natively interpret the reusable components we frontend developers love so much. We often think of frameworks like ReactJS or VueJS when componentizing our code, but what if we could write reusable components that were framework agnostic?

Next Generation Forms with React Final Form

Here at Eventbrite, we know the joys and pains of working with forms inside of ReactJS. When Erik Rasmussen(@erikras) hit the stage to offer one Form to rule them all, our ears perked up. The author behind Redux Form went back to the drawing board and iterated a new solution for forms. Unlike its predecessor, Redux Form, Final Form has no dependency on Redux and has a bundle size that’s about five times smaller. Final Form by itself is framework agnostic, but Rasmussen also provides a thin wrapper library for interfacing with ReactJS which works by taking advantage of React’s subscription utility.

Help! My React app is slowwwww!

Last but not least, our very own Principal Software Engineer, Ben Ilegbodu(@benmvp), hit the stage and got our blood pumping with a quick workout. Thought you’d have to skip your workout during the conference? Ben’s got you covered. It was just what the crowd needed before he brought us some insights he’s gathered while working on ReactJS right here at Eventbrite. As a premier ticketing platform, the last thing we want is for our website to feel slow for our users. Ben covers everything from breaking up component markup to combining Redux dispatch calls.

The best talk

Choosing from the long list of great talks is hard. If forced to pick one, I’d go with the talk that wasn’t on the official conference schedule: it was the conversations with my peers which brought up insights and questions inspired by the talks.

Eventbrite's Team at React Alicante

Where you there? Which talk was your favorite? What’s the most valuable part of attending a conference for you? Drop us some thoughts in the comments or ping me in Twitter @pixelbruja.

Getting started with Unit Tests

“No amount of testing can prove a software right, but a single test can prove a software wrong.”— Amir Ghahrai

Many developers think that Unit Testing is like flossing. Everybody knows it’s good, but many don’t do it. Some even think that with good code, tests are unnecessary. For a small project you are working on, that might be ok. You know the definition, the implementation details, and the desired behavior because you created them. For any other scenario, it’s a slippery slope, to say the least.

Keep on reading and learn why unit tests are an essential part of your development workflow and how you can start writing them for your new and legacy projects. For those who are unfamiliar to unit testing, you might want to start with a thoughtful article about what they are.

Example code

An unexpected morning regression

Last week I came into the office, grabbed my first coffee, leaned on my chair and start sipping from my old Tarantino’s cup while reading a bunch of emails, as usual. Eventually, I opened the code and faced what I had left undone the day before. “What was I thinking about?” I muttered as I started pounding the keyboard. “I managed to fix that!”

A few days later, we discovered a regression caused by that same line of code. Shame on me. “How could our unit tests allow this to happen?” Oops! No tests whatsoever, of any kind. Now, who’s to blame? Nobody in particular, of course. All of us are responsible for the codebase, even for the code we didn’t write, so it’s everybody’s fault. We need to prevent this from happening again. I usually forget what I broke — and, especially what I fix— these missing tests should be the first to start with.

Here are a few steps I should have followed before crashing our codebase first thing in the morning:

  • If I change any code, I am changing its definition and the expected behavior for any other parts involved. Unit tests are the definition. “What does this code do?” “Here are the tests, read them yourself.”
  • If I create new code, I am assuming it works not just for my current implementation, but for others to come. By testing, I force myself to make it extendable and parameterizable, allowing me to think about any possible input and output. If I have tests that cover my particular case, it is easy to cover the next ones. By testing, we realize how difficult it could be for others to extend our first implementation. This way, our teammates won’t need to alter its behavior: they will inject it!
  • If I write complex code, I ally encounter someone that puts me to the test: “Does this work?”, “Yes, here are the tests. It works”. Tests are proof that it works, your best friend and lawyer. Moreover, if someone messes up and your code is included somewhere, chances are developers summon you to illuminate the situation. Probably your tests will guide you to narrow the issue.
  • If I am making a new feature, I should code the bare minimum necessary for it to work. Writing tests first, before actually writing any real code is the fastest and most reliable way to accomplish that. I can estimate how much time I spend writing tests. I cannot estimate how long I will spend in front of the debugger trying to figure out where things went south because I made the whole thing a little too complicated.

Now I want to write unit tests. What’s next?

Let’s say I have convinced you that tests are not a dispensable part of our daily work, but your team does not believe in this. Maybe they think there is still too much work to do, of that if you were to write all the missing tests, that would take weeks, even months! How can you explain this to your Product Owner?

Here’s the thing: you won’t. If testing becomes a time-demanding task that requires it to be on a plan or a roadmap, it won’t likely ever take off. However, I want to offer you some tips to get started with testing that would work both if you have a significant deficit in test or you just started a new project:

  • Write unit tests first if you don’t know where to start.
  • Only write tests for the code you made and understand.
  • Don’t test everything. Just ensure that you add a couple of tests will every time you merge code.
  • Test one thing only. I’d rather maintain five simple tests than one complex one.
  • Test one level of abstraction. This means that when you test a component which affects others, you can ignore them. Make the component testable instead of testing everything around it.
  • If some new code is too complex to test, don’t. Rather, try to split it into smaller pieces and test each individually.
  • Don’t assume current locales or configuration. Run tests using different languages and time zones, for instance.
  • Keep them simple: Arrange just one “System Under Test” (SUT), perform some action on it to retrieve an output, and assert the result is the one you want.
  • Don’t import too much stuff into test suites. The fewer components involved, the easier it is to test yours.
  • Start testing the borders of the system, the tools, and utility libraries. Create compelling public APIs and test them. Ignore implementation details, as they are likely to change, and focus on input and outputs.

Remember, these tips work well for a codebase with no tests. The very first time you are about to fix, refactor or change the behavior of any part of the code, you must write the tests first to ensure you are not breaking anything. However, when working with legacy code, you would likely see the test coverage increase as the code changes.


In this blog post, we included some pieces of advice taken from our own experience with unit testing. There are other types of tests, but if you and your team want to start testing, unit tests suit you best.

Unit tests are more “straight to the point” than any other kind since they focus on validating single parts of a more complex codebase. If you are new to them, don’t panic: start from the smallest piece and build upon that. You’ll learn a lot along the process, and detect implicit dependencies or troublesome APIs you had previously skipped.

One nice thing about testing is that you make a massive leap towards coding from the outside out — instead of from the inside out, which is usually better for the implementer, and never for the user — which turns out to create a more elegant, comprehensive, and extendable code. It goes without saying that manual testing is still a thing.

What’s your experience with testing? Is there any other tip you would suggest to newcomers? Drop us some lines in the comments or ping me directly in Twitter @Maquert.

Photos by Markus Spiske and Isis França on Unsplash.

The “Aha” Moments of Becoming an Engineering Manager

Making the leap to being a manager is one of the most challenging transitions an individual contributor (IC) can choose to make in their career. And, let me tell you, my initial transition from IC to manager was not especially graceful — I would lovingly describe the two years it took me to figure things out as a time of “epic failure.”

A couple of months back, I shared this story on the main stage at ELEVATE, a San Francisco-based event for engineering leaders. In front of an audience of peers and senior engineers – some friendly faces, but mostly all strangers – I shared how dropping the ball as a new manager was, in fact, an invaluable lesson. The “aha” moment I experienced has stuck with me throughout my career as a manager, as a leader of leaders, and, today, as the SVP of Platform at Eventbrite.

Three key tips I share with engineers or technical folks thinking about (or struggling with) transitioning from IC to manager are:

  • Know how you work best and make adjustments (ex: I personally don’t multitask well).
  • Understand the skills that made you good at one level won’t necessarily translate into the next.
  • Learn how to embrace a growth mindset.

You can hear me expand on these points as my presentation was graciously captured on video and you can see it here:

Radical Transparency: Biggest Learnings From Transitioning to Management

About 10 minutes in, I flip to being the interviewer, navigating a fantastic discussion packed with insights between four esteemed panelists — Yi Huang, Sue Nallapeta, David Murray, and Hala Al-Adwan — who hold engineering leadership roles at Facebook, Zoosk, doctor.com, and Signal Science, respectively.

Ultimately, the lessons I learned as a result of my “epic failure” ended up shaping me into the engineering manager I am today and impacted the trajectory of my career. I hope others also benefit from my shared tale of woe when navigating their own transitions into the next level.

Have you experienced an “aha” moment in the face of a career challenge? Tell us about it in the comments.

Grace Hopper 2018: Five Unforgettable Experiences

I just came back from the Grace Hopper Celebration, where I joined another 22,000 women in the world largest gathering of women technologist. This year’s conference was outstanding, and I’m excited to share the standout moments that stayed with me the most. Keep on reading to learn about the most inspiring, surprising, outstanding and best career advice I got from the 2018 event.

I couldn’t believe I was going!

The first time I heard about Grace Hopper – the world’s largest gathering of women technologists produced by AnitaB.org – was a year ago. My VP at the time recommended I attend. Unfortunately, it was way too late to register, but I tuned into the live stream of the keynotes and felt inspired just by watching remotely. I immediately set a reminder on my calendar to register for the 2018 celebration as soon as I could – I was determined to be apart of the largest women’s conference in the world. On the day registration opened, I blocked off my calendar to refresh the site until the magic registration button appeared. When I got my ticket, it was almost surreal!

At this year’s conference, over 22,000 women descended on downtown Houston, TX to take part in celebrating women in tech. There were several learning tracks this year, and the most popular were artificial intelligence (AI), mentorship, and career development. There are five different session types – panel, workshop, presentation, mentoring circles, and poster sessions.

Most Inspiring

Jessica Matthews, CEO of Uncharted Power, gave an inspirational keynote about how she started a company that develops kinetic energy storing products. She immediately connected with the audience when she admitted that she strives to be the perfect love child of Beyonce and Bill Nye the Science Guy. When she spoke about how she got to this point in her career, her message was simple: “Just because it’s not your plan doesn’t mean it’s not your destiny.”

At the age of 19, her career started when she developed the product idea for a college class project. Over her lifetime she visited her family several times a year in Nigeria and noticed a severe infrastructural problem in the community. Villages and even cities would frequently lose power for extended periods of time and their solution during these periods was the required use of kerosene lamps or diesel generators, which poses a significant health hazard from the fumes.

For the class project, she developed a short-term solution. Her idea was to use a soccer ball to store kinetic energy as you play. When she started distribution of her new product, she learned that girls in refugee camps were not allowed to play soccer or play outside past the age of 12. To solve that challenge, she applied the same technology in the soccer ball to a jump rope. To the crowd’s delight, she started to jump rope in heels on stage to demonstrate how her product generated light.

Jessica is building the cities of tomorrow and building them faster, more cost-effectively, and more durable. Without having her background or feminine perspective, none of this would be possible. In fact, her background has enabled her to have a unique perspective, to think about solving problems differently.

Shattering the Mid-Management Barrier: Moving to an Executive Role

Best Career Advice

The session, “From Shattering the Mid-Management Barrier: Moving to an Executive Role,” included a panel of accomplished women in executive-level roles who shared how they made it. My key takeaways were to ‘take a risk’ and get sponsorship. Each one had a unique story to tell, and all of them either moved into a role where they had limited experience or had someone advocating for them to open up key opportunities.

Many women turn down an opportunity to grow because it seems risky or they don’t feel they have all of the qualifications. These experts say, “Go for it.” Start by asking for something small that you would never do. For example, one woman described asking for a discount for a recent purchase and she got the discount! One woman applied for a job she never felt qualified for but was invited for a job interview. In your career, it is about taking the right calculated risk.  

Most Surprising

Grace Hopper’s career expo was a sight to behold. It is considered the world’s largest career fair and did not disappoint. As you enter into the door, the displays resemble that of an amusement park with large signs, lights, and entertainment. Hundreds of recruiters were happily buzzing around their colorful and interactive booths with freebies ready to talk about their company. You could easily spend your entire week interviewing and take home an entire suitcase full of swag.

As you pass by each booth, you can talk to one of the available recruiters about potential openings as you would at a typical career fair. If there is mutual interest, an interview will be scheduled later the same day. If interviewing is your intention when attending Grace Hopper, you can upload your resume to the Grace Hopper resume database. Potential employers will call or send emails to set up interviews while you’re both onsite. Companies are so eager to find talent; two attendees received job offers on the spot.

Most Valuable Session

At the “How to Have a Successful Grace Hopper Celebration 2019 Talk Submission” session, I learned that only 24% of the average 2400 submissions would get accepted. This rate hit close to home as I submitted a presentation for this year that was not accepted, but after attending this session, I have a new strategy. Here’s a summary of what I learned:

  • Follow the Rules. There are strict guidelines for how you do your writeup from word count to page number
  • Pick the Right Track. The track is most important! You must choose a track that aligns to the content you want to present. Pro tip: Presentation or panel formats are ideal for first-timers.
  • Content that Aligns with Conference Themes. Career, mentorship, and AI stood out as essential themes for this year. Excited to see what will be hot for 2019.
  • A Clear Write Up of Your Proposal. You must provide a catchy title, abstract, and detailed information to get your talk accepted. Don’t leave out specific details in fear of revealing too much.
  • Target Areas for Low Submissions. The organizers need more submissions for senior and executive women sessions. There is also a need for more advanced technical topics. Submitting in these areas help increase your chance of getting accepted.

Most Outstanding Experience

The best experience was meeting people. I met women from all over the world – DC, the Bay Area, Boston, and even Ireland. I’ll stay in touch with all of them. If either of us is in the same city, we each have an open invitation to meet in person.

While the celebration is over, the inspirational message and excitement remain. I’ll take what I have learned and inspire others at my company and in my community. It is never too early to start planning for next year’s conference on October 2 – 4 in Orlando, Florida. Will I see you there?

The Lies We Tell Ourselves

A couple of years ago I had an opportunity to be promoted. I evaluated the role. I had a plan for how I would implement in that role. I talked to the hiring manager (my manager at the time). I talked to my network inside the company. I knew the competition and, in my estimation, I was a better fit for the role and for the direction the organization should take. I was certain I had the position in the bag.

And then, I didn’t get the promotion. One of my peers got the position and became my new boss. I was devastated, humbled, and felt rejected by the organization. Looking back now, with the benefit of hindsight (and many conversations while I was still employed at that organization), I know that I was suffering from a few key self-deceptions. I could not see the possibility of failure. I only talked with people in my network who would be most likely to confirm my assumptions. I left no room in my mind for doubt. Doing so, I blocked out the possibility of addressing any weaknesses I presented to the hiring manager.

At that moment, when I didn’t get promoted, I was faced with cognitive dissonance. Cognitive dissonance occurs when our beliefs or ideas don’t match our actions or reality. The result is emotional discomfort that we then attempt to resolve.  Often, the resolution comes to us through further self-deception. I believed I was going to get the job and then the reality didn’t meet my beliefs. What followed were additional self-deceptions to mitigate the weight of heavy emotions and soothe the pain I was feeling. One of the greatest gifts of this experience was the ability to look back and identify ways I self-deceive and the further turmoil those behaviors cause.

In truth, we all struggle through moments of cognitive dissonance. Sometimes we are aware of it, and most of the time we aren’t. To create cognitive harmony, we usually engage in self-deceptive strategies. Sometimes, our self-deceptions are incredibly useful. For example, when we engage in self-enhancing behaviors like “fake it ‘til you make it” to begin to adopt a new habit. More often, self-deception is toxic. Like when organizations believe their plans are truth despite data that shows we never deliver work when we say we will, which results in a myriad of poor behaviors throughout organizations.

Self-deception is subtle and hard to spot in ourselves. The rest of this post will discuss:

  • How to identify self-deceptions and why we use them.
  • What impact these behaviors can have on individuals, teams, and organizations.
  • Strategies for consciously deciding to engage or not engage in self-deception.

Identifying Self-Deceptions

“Knowing is half the battle” ~G.I.Joe

Willful Ignorance

Willful ignorance is a choice not to see information that contradicts our core beliefs. Every unresearched Facebook debate or meme provides an opportunity to identify this particular self-deception.

At work, this looks more like picking a deadline without validating it against empirical evidence or without leveraging the wisdom of the teams doing the work. It can also be things like believing you are the best team member (or worst) without asking your teammates for feedback.

Reality Denial

Reality denial has a lot of overlap with willful ignorance. They are close cousins. Whereas willful ignorance is avoiding the consumption of information that conflicts with your beliefs, reality denial is refuting new information because it flies in the face of what you want or believe.

Example 1

Even though we haven’t made our commitments the last three sprints (or iterations), we will continue to commit to more than we have completed because we believe we can or should do more.

Example 2

Though everything the team is telling me indicates that we will not meet the deadlines, as a leader, I will continue to insist that we still have to meet the deadline without any change to scope because I believe this is the only way we can succeed.

Example 3

As an organization, we design open space offices because we believe our employees will be more productive, despite the data that currently indicates a reduction in productivity in open environments.


Overconfidence is a lie we tell ourselves to elevate ourselves in our own perception. The maxim “fake it til you make it” is a prime example of this sort of self-deception. It can have significant benefits in that it helps us do things like public speaking or stretching our skills. It can also cause us to underestimate risks and difficulties in our efforts. My opening story was filled with overconfidence. The result was that I did not prepare the way I should have for that role and those around me easily saw that lack of preparation.


Self-enhancing deceptions are a close cousin to overconfidence. In this case, we enhance ourselves to prove to others how awesome we are. Blustering, exaggerating, and one-upping tend to live here. Also in this space is developing one’s brand or self-promotion to move ahead in our life or career.


On the other side of the spectrum, we have self-handicapping.  Insecurity and imposter syndrome can play an essential role in perpetuating this deception. We use it to protect ourselves from the perception of inevitable failure due to our belief we lack the skills or knowledge to reach our goals. For example, not applying for a job or promotion that is perhaps a stretch assuming there’s no way you could get the role. The worst that could happen is you don’t get the role which means nothing changes. You might even walk away with a clearer understanding of what work you need to do. Alternatively, you could get the role. There’s only one way to know.

Confirmation Bias

Confirmation Bias is the act of ingesting all the facts but only seeing those that support or confirm what we already believe. In addition to helping avoid cognitive dissonance, we employ this self-deception when there’s too much information to sift through. This can show up in offices when we only select articles that support our viewpoints to make our points with colleagues. We can also see this in the people we choose to spend time with by creating an echo chamber of agreeable opinions. The latter is particularly troublesome if the person creating the echo chamber is in leadership and is only hiring and promoting people that agree with their choices and vision.

Self-serving Bias

Self-serving bias is when we credit our success to our hard work and our failures to external factors.  In my opening example, I allowed a self-serving bias in my thinking to soothe my hurt feelings. The thoughts in my head went something like this:

  • “I didn’t get the promotion because my boss didn’t provide me with enough information to prepare for the role aptly.”
  • “My boss didn’t communicate what he perceived as promotion blocking flaws in our many conversations so that I could address them and improve.”

In retrospect, I’m pretty sure my boss tried to communicate these things, and I was so blinded by over-confidence and self-enhancement that I didn’t see the reality of the situation.


Like self-serving bias, attribution error is a way to excuse someone else’s bad behavior by focusing on dispositional factors. The belief that what people do is representative of who people are is what prompts this error. For example, if a colleague didn’t make a deadline you might say:

  • “Well, he hasn’t learned the skills necessary yet.”
  • “She’s just lazy.”
  • “He’s a slacker.”

Often this self-deception is used to explain a person’s behavior when it creates discomfort for us. “Well, that’s just the way he or she is” can be much more easily brushed off without ever confronting the other person, which creates conflict.

The Impact of Self-Deception

Negative Impacts

As I’ve led individuals and teams through self-discovery, asking them to name their top values, usually they include one of these: truth, honesty, trust, or authenticity. Self-deception, at its core, is a lie. The very act of self-deception is against most people’s desire for truth. Generally, other people are better at seeing reality than the person in the throes of self-deception. This interplay undermines trust between individuals, on teams, and inside organizations. Trust is necessary for psychological safety. Psychological safety is necessary for high productivity and effectiveness.

Also, in individuals, the ability to grow is stunted. Either they do not see their flaws or they do not see their strengths. Both issues undermine someone’s ability to grow, change, and improve. In leaders, this can be exacerbated since organizations can only grow as far as their leaders have grown.

Positive Impacts

It’s not all bad though! Sometimes, self-deception is just what we need to be a little more humble or to spur us to act courageously. A leader in an organization might put on a little bravado about the vision they’re painting to inspire a company to greatness. An individual might enhance herself through her personal brand to find a new job.

It comes down to self-awareness, choice, and a little bit of humbleness to encourage the positive outcomes of self-deception and discourage the negative ones.

Choosing Right Action over Passive Self-Deception

Practice Self Awareness

Self-awareness is a mental muscle. Like other muscles, we can exercise and make it stronger over time. Here are some questions you can use to evaluate how much self-deception you are allowing:

  • What behaviors am I exhibiting that undermine what I say or believe?
  • What makes me nervous or scared about this situation?
  • What can I do to validate the internal stories that are fueling my fears?
  • What self-deception most resonates with my current situation?
  • How is this deception helping me? How is this deception hurting me?
  • What impact does this deception have on those around me?

You may not be able to catch yourself in the moment. Sometimes looking back at a situation can be enough. As you evaluate the self-deceptions we covered, was there one that brought forward a clear example for you? Leverage that experience and hold these questions up to it. Often, our self-deceptions are well-worn paths of habit. By looking at your patterns in one situation, you can find your patterns in the present and future.

Have a Support System

Self-awareness is a great tool, but it shouldn’t be your only tool in the toolbox. It’s important to get out of your box and leverage those around you. Get out of your echo chamber.  Welcome opinions that are uncomfortable. Surround yourself with people who challenge you to question your beliefs, values, and intentions.

Once you do that, then you have to listen. Be ready to be challenged. To do that takes vulnerability and it is easy to fall back into self-deception in the midst of dropping our guard. It will feel unsafe. It is easy to become defensive and leverage self-deceptions to protect your perception of self or diminish negative emotions.

Go courageously into the midst of your closest friends and loved ones anyway.

How Your Company Can Support Junior Engineers

My first day as a junior software engineer was one of the most mentally draining days of my life. The next three months on the job weren’t much better. I didn’t leave my house except to go to work or attend a work event. I remember thinking, What did I sign up for?

Read on to learn what your company needs to provide to avoid overwhelming situations for junior engineers and to make it easy for them to be a contributing member of your organization.

Being a junior engineer

I had been coding since I was twelve, but coding was fun. Working as a software engineer was something else. I felt like everyone else in the room was reading Shakespeare and I was trying to get through a copy of Goodnight Moon.

I was the first junior engineer that my company hired. Ticketfly – now part of Eventbrite – had a fantastic attitude about onboarding a junior hire, but it was still tough. My mentor built out an incredible Ember.js app where I had to solve small (I sure didn’t think they were small at the time) problems in JavaScript to make the tests pass. After that, I worked on a small app for fun applying the same framework the company used.

Ticketfly made a meaningful investment in my career, and also took a significant risk in hiring a new junior engineer. However, it paid off for them – and for me. I stayed with Ticketfly through two acquisitions and was promoted to senior software engineer. Also, I’m now a senior software engineer at Eventbrite.

With the lack of senior engineers available in the hiring pool, many companies are now hiring junior engineers – often hiring them right out of coding bootcamps – or creating apprenticeship programs within the company. This is a smart strategy to add talent to an engineering team, but while hiring an experienced or senior engineer adds a contributing member to your team, hiring a junior engineer is a long-term investment.

If companies rush to hire junior engineers before determining the company has the support system, resources, senior team members willing to mentor new engineers, and a realistic timeline for when a junior engineer will add value to the team – everyone suffers.

In the next sections I’ll describe some of the elements necessary for a robust junior engineering training program at your company:


At Eventbrite, we have outstanding support systems in place for our junior engineers. It starts at an orientation where you’re assigned a ‘buddy’ who helps you with anything from finding your way around the office to helping you set up your development environment. Eventbrite runs a mentorship program where we match mentees to a mentor based on what they want to learn. We also host weekly Code Labs where senior engineers teach a new topic each week. The Code Lab is a comfortable environment for the junior to ask questions without feeling incompetent.

Take away

Do you have a knowledgeable engineer willing to take the time to mentor and pair with the new hire? Onboarding and teaching a junior engineer is a considerable responsibility. It could take months before they’re entirely independent. It’s beneficial to determine whether you can afford to hire someone who might take months to ramp up.

Managing expectations

One of the incredible things about my current company is its focus on jumpstarting the careers of junior engineers. Eventbrite has hired more bootcamp graduates from Hackbright (an all women coding bootcamp) than any other company out there.

Some of our junior hires are the best engineers I’ve ever worked with. While each new junior hire took months to bring up to speed and train – the investment we’ve made in hiring more junior developers has paid off for Eventbrite. Most of the hires are still at the company years later, and are exceptional contributors to the codebase.

Take away

Before hiring a new junior engineer, think about your timing. Would you have time to mentor this engineer?

Pair programming

Having a culture of pair programming is a great benefit when hiring a junior engineer. One of the great things for me when I started out my career was that my mentor would pair with me when I got stuck on an issue, but he also encouraged me to go figure things out on my own. It was a great mix because I had support so I wouldn’t feel stressed, but I also was given the independence to learn and grow.

We practice driving and navigating pairing at Eventbrite. It helps to keep both members of a pair actively engaged in solving the problem. Based on my experience, it’s useful to hire several junior engineers at a time so they can pair together and support each other. It’s also helpful to the company because you can onboard and train them at the same time.

Take away

Try assigning an easy task to the junior engineer to take a stab at it first. If they run into problems, try pairing on the problem. Eventbrite buddies have regular check-ins with new hires to make sure things are running smoothly.


Eventbrite hosts an incredible training session on React with our principal Front-End engineer, Ben Ilegbodu. The session walks you through building out an application in React and provides solutions in separate branches if you get stuck so that anyone can keep up with the pace. It’s incredibly welcoming and inclusive when you’re just starting out.

Most companies tailor their onboarding programs toward engineers with years of experience. This practice can be frustrating to a junior engineer. One of the great things that Eventbrite does to support junior engineers is to provide a buddy just for setting up your dev environment.

Another side of it could be online training. Eventbrite provides Udemy accounts for all employees. I’m continually taking online courses, and it’s fantastic to work at a company that supports continuing your education.

Take away

Creating a small repository with tests that your junior engineer needs to get passing can get someone onboarded quickly. Consider finding a training program if your company doesn’t have the resources to create one internally. Find an interesting conference that offers a workshop.

Wrapping up

My first company took a chance by hiring a junior engineer and I’ll always appreciate they made that commitment. While I was their first junior engineer, they had a strategy, realistic expectations, provided me with a strong support system, and invested in my learning. That’s what it takes to support a junior engineer that has a shot at becoming an asset to your team.

Do you remember your first day as an engineer? Share your funny junior engineering mistake in the comments below or reach out to me on Twitter @randallkanna.

Creating Flexible and Reusable React File Uploaders

The Event Creation team at Eventbrite needed a React based image uploader that would provide flexibility while presenting a straightforward user interface. The image uploader components ought to work in a variety of scenarios as reusable parts that could be composed differently as needs arose. Read on to see how we solved this problem.

What’s a File Uploader?

In the past, if you wanted to get a file from your web users, you had to use a “file” input type. This approach was limited in many ways, most prominently in that it’s an input: data is only transmitted when you submit the form, so users don’t have an opportunity to see feedback before or during upload.

With that in mind, the React uploaders we will be talking about aren’t form inputs; they’re “immediate transport tools.” The user chooses a file, the uploader transports it to a remote server, then receives a response with some unique identifier. That identifier is then immediately associated with a database record or put into a hidden form field.

This new strategy provides tremendous flexibility over traditional upload processes. For example, decoupling file transportation from form submissions enable us to upload directly to third-party storage (like Amazon S3) without sending the files through our servers.

The tradeoff for this flexibility is complexity; drag-and-drop file uploaders are complex beasts. We also needed our React uploader to be straightforward and usable. Finding a path to provide both flexibility and ease-of-use was no easy task.

Identifying Responsibilities

Establishing the responsibilities of an uploader seems easy… it uploads, right? Well sure, but there are a lot of other things involved to make that happen:

  • It must have a drop zone that changes based on user interaction. If the user drags a file over it, it should indicate this change in state.
  • What if our users can’t drag files? Maybe they have accessibility considerations or maybe they’re trying to upload from their phone. Either way, our uploader must show a file chooser when the user clicks or taps.
  • It must validate the chosen file to ensure it’s an acceptable type and size.
  • Once a file is picked, it should show a preview of that file while it uploads.
  • It should give meaningful feedback to the user as it’s uploading, like a progress bar or a loading graphic that communicates that something is happening.
  • Also, what if it fails? It must show a meaningful error to the user so they know to try again (or give up).
  • Oh, and it actually has to upload the file.

These responsibilities are just a short list, but you get the idea, they can get complicated very quickly. Moreover, while uploading images is our primary use case, there could be a variety of needs for file uploading. If you’ve gone to all the trouble to figure out the drag / drop / highlight / validate / transport / success / failure behavior, why write it all again when you suddenly need to upload CSVs for that one report?

So, how can we structure our React Image Uploader to get maximum flexibility and reusability?

Separation of Concerns

In the following diagram you can see an overview of our intended approach. Don’t worry if it seems complicated -we’ll dig into each of these components below to see details about their purpose and role.

React Architecture illustration showing overview of several components that will be enumerated below.

Our React based component library shouldn’t have to know how our APIs work. Keeping this logic separate has the added benefit of reusability; different products aren’t locked into a single API or even a single style of API. Instead, they can reuse as much or as little as they need.

Even within presentational components, there’s opportunity to separate function from presentation. So we took our list of responsibilities and created a stack of components, ranging from most general at the bottom to most specific at the top.

Foundational Components

Recap of the architecture illustration from above with "Foundational Components" highlighted.


This component is the heart of the uploader UI, where action begins. It handles drag/drop events as well as click-to-browse. It has no state itself, only knowing how to normalize and react (see what I did there?) to certain user actions. It accepts callbacks as props so it can tell its implementer when things happen.

Illustration showing a clear pane representing UploaderDropzone over a simple layout in React

It listens for files to be dragged over it, then invokes a callback. Similarly when a file is chosen, either through drag/drop or click-to-browse, it invokes another callback with the JS file object.

It has one of those inputs of type “file” that I mentioned earlier hidden inside for users who can’t (or prefer not to) drag files. This functionality is important, and by abstracting it here, components that use the dropzone don’t have to think about how the file was chosen.

The following is an example of UploaderDropzone using React:

    Drag a file here!
    <Icon type="upload" />

UploaderDropzone has very little opinion about how it looks, and so has only minimal styling. For example, some browsers treat drag events differently when they occur on deep descendants of the target node. To address this problem the dropzone uses a single transparent div to cover all its descendants. This provides the needed experience for users that drag/drop, but also maintains accessibility for screen readers and other assistive technologies.


The UploaderLayoutManager component handles most state transitions and knows which layout should be displayed for each step of the process, while accepting other React Components as props for each step.
Illustration showing a block representing the UploaderLayoutManager and several smaller blocks representing layouts being flipped/shuffled

This allows implementers to think about each step as a separate visual idea without the concern of how and when each transition happens. Supporters of this React component only have to think about which layout should be visible at a given time based on state, not how files are populated or how the layout should look.

Here is a list of steps that can be provided to the LayoutManager as props:

  • Steps managed by LayoutManager:
    • Unpopulated – an empty dropzone with a call-to-action (“Upload a great image!”)
    • File dragged over window but not over dropzone (“Drop that file here!”)
    • File dragged over dropzone (“Drop it now!”)
    • File upload in progress (“Hang on, I’m sending it…”)
  • Step managed by component that implements LayoutManager:
    • File has uploaded and is populated. For our image uploader, this is a preview of the image with a “Remove” button.

The LayoutManager itself has little or no styles, and only displays visuals that have been passed as props. It’s responsible for maintaining which step in the process the user has reached and displaying some content for that step.

The only layout step that’s externally managed is “Preview” (whether the Uploader has an image populated). This is because the implementing component needs to define the state in which the uploader starts. For example, if the user has previously uploaded an image, we want to show that image when they return to the page.

Example of LayoutManager use:

    dropzoneElement={<DropzoneLayout />}
    windowDragDropzoneElement={<WindowDragDropzoneLayout />}
    dragDropzoneElement={<DragDropzoneLayout />}
    loadingElement={<LoadingLayout />}
    previewElement={<PreviewLayout file={file} />}



Resource-Specific Components

Recap of the React architecture illustration from above with "General Components" highlighted.


The ImageUploader component is geared almost entirely toward presentation; defining the look and feel of each step and passing them as props into an UploadLayoutManager. This is also a great place to do validation (file type, file size, etc).

Supporters of this tool can focus almost entirely on the visual look of the uploader. This component maintains very little logic since state transitions are handled by the UploaderLayoutManager. We can change the visuals fluidly with very little concern about damaging the function of the uploader.

Example ImageUploader:

const DropzoneLayout = () => (
    <p>Drag a file here or click to browse</p>
const DragDropzoneLayout = () => (
    <p>Drop file now!</p>
const LoadingLayout = () => (
    <p>Please wait, loading...</p>
const PreviewLayout = ({file, onRemove}) => (
        <p>Name: {file.name}</p>
        <Button onClick={onRemove}>Remove file</Button>
class ImageUploader extends React.Component {
    state = {file: undefined};

    _handleRemove = () => this.setState({file: undefined});

    _handleReceiveFile = (file) => {

        return new Promise((resolve, reject) => {
            // upload the file!
        .catch(() => this.setState({file: undefined}))

    render() {
        let {file} = this.state;
        let preview;

        if (file) {
            preview = (
                <PreviewLayout file={file} onRemove={this._handleRemove} />

        return (
                dropzoneElement={<DropzoneLayout />}
                dragDropzoneElement={<DragDropzoneLayout />}
                loadingElement={<LoadingLayout />}

Application-Specific Layer

Recap of the React architecture illustration from above with "Application-specific layer" highlighted.

The example above has one prominent aspect that isn’t about presentation:  the file transport that happens in _handleReceiveFile. We want this ImageUploader component to live in our component library and be decoupled from API specific behavior, so we need to remove that. Thankfully, it’s as simple as accepting a function via props that returns a promise that resolves when upload is complete.

_handleReceiveFile(file) {
    // could do file validation here before transport. If file fails validation, return a rejected promise.
    let {uploadImage} = this.props;


    return uploadImage(file)
        .catch(() => this.setState({file: undefined}))

With this small change, this same image uploader can be used for a variety of applications. One part of your application can upload images directly to a third party (like Amazon S3), while another can upload to a local server for a totally different purpose and handling, but using the same visual presentation.

And now because all that complexity is compartmentalized into each component, the ImageUploader has a very clean implementation:

<ImageUploader uploadImage={S3ImageUploadApi} />

With this foundation applications can use this same ImageUploader in a variety of ways. We’ve provided the flexibility we want while keeping the API clean and simple. New wrappers can be built upon UploadLayoutManager to handle other file types or new layouts.

In Closing

Imagine image uploaders which were purpose-built for each scenario, but contain only a few simple components made of presentational markup.  They can each use the same upload functionality if it makes sense, but with a totally different presentation. Or flip that idea around, using the same uploader visuals but with totally different API interfaces.

In what other ways would you use these foundational components? What other uploaders would you build? The sky’s the limit if you take the time to build reusable components.

8 Reasons Why Manual Testing is Still Important

The increase of test automation adoption has unjustly framed manual testing as an archaic and unnecessary practice. After watching an automation suite swiftly execute an entirely library of test cases, it can be easy to tunnel vision on the great benefits of automation. However, the value of manually executing your tests cannot be understated; here are a few reasons why manual is still relevant as ever.

Tape 1: Cycle Times

There’s no way around it; initial automation requires an increased investment in both, time & resources. You are setting up a foundation to continually benefit from in your future testing endeavors. However, in some cases, your automation efforts will not be the ideal solution for your testing.  Attempting to initialize automation while close to the end of your testing cycle would be a moot effort; the time you take to set up (and the sudden resource shift) means you’ll be nearing your release date before you can start running reliable and core automated testing. During that same timeframe, you could be focusing your testing resources towards manual execution. As the majority of their time is focused on test case validation, the end result is more coverage within your test cycle.

Tape 2: Even Your Automation Has Errors

Like any piece of code, your automation will contain errors (and fail). An error filled automation script may be misinterpreted as failed functionality in your tested application, or (even worse) your automation script will interpret an error as a correct functionality. Manually testing your core, critical-path functionality ensures that your test case is passing from a user perspective, with no room for misinterpretation.

Tape 3: UI Validations

The advent of automated testing platforms for Responsive and UI testing has provided a much appreciated convenience. However, it should be a boost to your UI testing efforts, not a crutch. These programs validate your test cases by checking element distance, image placement, and alignment of elements in relation to each other. Because of this, there are more than a dozen ways that something such as alignment between a menu and logo can be misinterpreted; a manual tester would immediately be able to catch something that looked “off”, and fail the test case.

Tape 4: Un-Automatable Scenarios:

Some scenarios are simply not feasible to automate; they are either actually impossible due to technological limitation + the complexity of the scenario, or the resource cost of automating it greatly outweighs the cost of a simple manual test. Case in point, we recently had a customer who needed to test their manual tap-and-pay function for their mobile wallet app. Developing a way to automate this scenario is not worth it when compared to manually testing it with your device.

Tape 5: (Short-Term) Cost

Over time, automation leads to cost savings, faster execution, and continous testing. In the immediate short term however, there is an investment cost (and learning curve for the unfamiliar) that can be a situational disadvantage. The cost of setting up and running your initial automation framework can range anywhere from 5-15x the cost of your manual testing endeavors. And as discussed earlier, implementing automation while crunched for time towards the end of a test cycle will not allow you to enjoy automation’s full potential. Choosing to conduct manual testing at this stage provides an immediate, tangible result from your testing resources.

Tape 6: Exploratory Testing

Exploratory testing describes the process of freely testing the application for the purpose of finding defects can`t subsequently designing new test cases. Defects found through exploratory testing are often the results of testing complex scenarios that would not have been addressed through your predefined test cases. Having a foundation of core, repeatable tests automated will free up time to designate resources towards exploratory testing.

Tape 7: Skills

While the end result of Automation is ease, the set up of framework and development of scripts are no easy tasks. An effective automator has a foundation of programming skills, as well as an inherent understanding of test design. These skills are learned over years of experience in both QA and Development, and acquiring somebody with these specific skillsets (especially on short notice) is not a simple process. On the other hand, the majority of Manual test cases are simple to execute and can easily be taught; follow the steps in the test case, and validate that your actual results are consistent with the expected results.

Tape 8: Agile

In the context of Agile testing, automation is of great benefit. Having a library of tests reliably and quickly executable truly helps with test completion & coverage during a tight sprint. By that same token, manual testing is a quick way to execute for any test cases that are not yet automated. There may be no time to build automation for new features introduced in the current build, making manual the best option for test completion.

As a conclusion, the need for increased test coverage across an ever increasing range of software and devices has made test automation more important than ever. As automation continues to grow, it can be easy to forget about the wide spectrum of benefits manual testing still has to offer. Appreciating the value of both approaches will make for a wholesome testing experience.

Looking under the hood of the Eventbrite data pipeline!

Eventbrite’s mission is to bring the world together through live experiences. To achieve this goal, Eventbrite relies on data-driven decisions at every level. In this post, we explore Eventbrite’s treasure trove of data and how we leverage it to push the company forward. We also take a closer look at some of the data challenges we’ve faced and how we’re pushing forward with new improvements to address these challenges!

The Data Engineering team at Eventbrite ingests data from a number of different sources into a central repository.  This repository is the foundation for Eventbrite’s analytics platform. It empowers Eventbrite’s engineers, analysts, and data scientists to create data-driven solutions such as predictive analysis, algorithmic newsletters, tagging/clustering algorithms, high-value customer identification, and search/recommendations.

The Situation: Degrading performance and increasing cost of existing Data Warehouse running on Hadoop infrastructure (CDH5)

We use MySQL as our main production datastore, and it supported most of our data analytics/reporting until a few years ago. Then we implemented a Cloudera CDH ecosystem, starting with CDH3 and upgrading to CDH5 when it was released. Prior to phasing in our CDH environment, our main OLTP (online transaction processing) databases were also powering our reporting needs.

Our production MySQL topology consists of a Primary-Secondary setup, with the majority of the read-only traffic directed to the MySQL secondary instances. For many years, we met our reporting requirements by leveraging the read-only MySQL instances but it came at a steep price due to contention and performance issues caused by long-running SQL queries.

As a result, we moved much of our reporting to our new CDH environment, and we designed a new set of transformation tables to simplify the data access  for Engineers, Analysts and Business users. It’s served us well as the backbone for our Data Warehouse efforts, but the time had come to take the next step as we’ve faced a number of challenges:


Our CDH5 cluster lives on Reserved Instances, and all of the data in the cluster is housed on local solid state drives.  As a result, the cluster is expensive to maintain.

A Reserved Instance is a reservation of resources for an agreed upon period of time.  Unlike on-demand, when you purchase an RI (reserve instance), you commit to paying for all  the hours of the 1-year or 3-year term. The end result is a lower hourly rate, but the long term costs can really add up.


We have a large collection of uncurated data, and we had not transformed the data into a single source-of-truth about our business. As a result, core business metrics (such as organizer, consumer, and event data) were reported differently in different places in the organization, and attributes such as currency, location and timezone were reported differently across business units.


Most jobs were scheduled via Oozie, there was little effective monitoring in place, and there was no method to track or enforce dependencies between coordinators. In addition, other analytics jobs that utilize Salesforce and MySQL data were scheduled through a local Windows machine that was prone to errors and regularly failed without warning or notification.


All ETL-processing and  ad-hoc queries executed on the same CDH5 cluster. Each process had its own load profile, so the cluster was configured to fit an aggregate of those loads. The end result was that jobs frequently conflicted with each other and competed for resources.

Our workload required burst capacity to support experimental development, ad-hoc queries, and routine ingestion scripts. In an ideal setup, we would scale up and scale down computing resources without any interruptions or data loss.


For MySQL ingestion, we used a home-grown wrapper called Sqoozie to integrate with our MySQL databases. Sqoozie combines Apache Sqoop – a command-line application for transferring data between relational databases and Hadoop – and Apache Oozie, a Hadoop workflow scheduler. It allows for writing MySQL tables directly to Hive tables. While this approach worked for smaller datasets, it became prohibitive as our data grew. Unfortunately, it was setup as a full ingestion of all tables each day and typically took most of a day to finish, putting high load on the shared resource cluster for an extended period of time.

For web analytics ingestion, we used a proprietary tool called Blammo-Kafka that pulled the web logs directly from Kafka daily and dumped them to Hive tables partitioned by day.

For Salesforce ingestion, we used the Salesforce Bulk API to ingest all objects daily and overwrite the previous day’s ingestion.

The Solution: EMR, Presto, Hive, and Luigi to the rescue!

In the past year, we’ve invested heavily in building a shiny new “data-foundry” ecosystem to alleviate many of the pain points from our previous CDH environment. It is the result of many whiteboard sessions, sleepless nights, and walks around the block at Eventbrite’s offices at our SOMA location in San Francisco and Cummins Station in Nashville.

We focused not only on improving stability and cost, but also on designing a new set of transformation tables that would become the canonical source-of-truth at the company level. This involved meeting with key stakeholders to understand business metrics and exploring new technologies. The following diagram depicts sample output from some of our working sessions. As you can tell, it was a tedious process.

The end result was the implementation of a new “data-foundry” infrastructure. The following diagram shows a general layout:

EMR (Elastic MapReduce) Clusters

Ingestion and ETL jobs run on daily and hourly scheduled EMR clusters with access to most Hadoop tools. Amazon’s EMR is a managed cluster platform that simplifies running big data frameworks such as Hadoop, Spark, Presto, and other applications in the Apache/Hadoop stack.

The EMR/S3 solution decouples storage from compute. You only pay for compute when you use it (high utilization). Multiple EMR clusters can access the data (S3, Hive Metastore), and interactive workloads (Hive, Presto, Spark) can be launched via on-demand clusters.

We’ve seen some benefits with Amazon EMR:

Intelligent resizing

  • Incrementally scale up (add nodes to EMR cluster) based on available capacity
  • Wait for work to complete before resizing down (removing nodes from EMR cluster)
  • Can scale core nodes and HDFS as well as task nodes

Cost Savings

By moving to EMR and S3, we’ve been able to considerably cut costs. With S3 we pay only for the storage that we use, not for total capacity. And with EMR, we’re able to take advantage of  “on-demand” pricing, paying low hourly rates for clusters only when we need the capacity. Also, we’ve reduced the cost even further by purchasing Reserved Instances and bidding on Spot instances.

  • Use Amazon EC2 spot instances to save > 80%
  • Use Amazon EC2 Reserved Instances for steady workloads

Reliability/Improved Operational Support

Amazon EMR monitors nodes in each cluster and automatically terminates and replaces an instance if there is a failure. Plus the new environment has been built from scratch, is configured via Terraform, and uses automated Ansible templates.

Job Scheduling

We use Luigi to orchestrate our Python jobs. Luigi enables us to easily define task workflows without having to know much about other workflows. It is an open source Python framework created by Spotify for managing data processing jobs, and it is really good at dependency management, which makes it a perfect tool for coalescing dependent data sources.

Centralized Hive Metastore

We have a centralized Hive metastore that saves all the structure information of the various tables, columns, and partitions for our Hive metadata. We chose Hive for most of our Hadoop jobs primarily because the SQL interface is simple. It is much cleaner than listing files in a directory to determine what output exists, and is also much faster and consistent because it’s backed by MySQL/RDS. This is particularly important since we rely on S3, which is slow at listing files and is prone to “eventual” consistency issues.


We continue to ingest production data from MySQL tables on a daily basis using Apache Sqoop, but in the “data-foundry” ecosystem we ingest the tables incrementally using “changed” columns to allow for quicker updates.

We ingest web analytics data by using Pinterest Secor to dump data from Kafka to S3. We then process it from that S3 path using Spark, both hourly and daily. Hourly we  ingest the latest data for each web analytics table since the last time it was ingested and write it to Hive tables partitioned by day and hour. Daily we also ingest the web analytics data to day partitioned Hive tables.

We ingest Salesforce data using a combination of the Salesforce REST and Bulk APIs using custom internal built Python clients for both. Tables are ingested through Spark using the API that makes the most sense based on the size of the data. Also, where available, we use primary key chunking in the Bulk API to optimize ingestion of large tables.

In addition to the ingestion processes that bring us to feature parity with the old CDH5 infrastructure, we also ingest data from a few other sources, including Google Analytics and several other 3rd party services.

We ingest Google Analytics data three times a day for the current day and once for the previous day based on SLAs provided by Google. We use Spark in addition to Google’s BigQuery and Cloud Storage clients to ingest Google Analytics data for our mobile app, organizer app, and web app to Hive tables partitioned by day.


By separating analytics processing from visualization and queries, we’ve been able to explore more tooling options. Both Presto and Superset have proven to be useful.  

Presto is a distributed SQL query engine optimized for ad-hoc analysis. It supports the ANSI SQL standard, including complex queries, aggregations, and joins. Presto can run on multiple data sources, including Amazon S3. We’re using Presto with EC2 Auto Scaling Groups to dynamically scale based on usage patterns.

Presto’s execution framework is fundamentally different from that of Hive/MapReduce. It has a custom query and execution engine where the stages of execution are pipelined, similar to a directed acyclic graph (DAG), and all processing occurs in memory to reduce disk I/O. This pipelined execution model can run multiple stages in parallel, and it streams data from one stage to another as the data becomes available. This reduces end-to-end latency, and we’ve found Presto to be quite snappy for ad-hoc data exploration over large datasets.

An additional benefit is that Facebook and the open-source community are actively developing Presto, which has no vendor lock-in because it speaks ANSI-SQL.

Superset is a data exploration and visualization tool that was open sourced by Airbnb. It allows for fast and flexible data access and comes complete with a rich SQL IDE, which is used heavily by Eventbrite’s business analysts.


We’ve introduced a new set of staging tables in our data warehouse that transform the raw data into dimension tables aligned specifically to meet business requirements.  These new tables enable analytics, data science, and reporting. The goal is to create a single “source-of-truth” for company metrics and company business concepts.

Data Exports

The Data Engineering team has developed a set of exporter jobs in Python to push data to targets such as Redis, Elasticsearch, Amazon S3 or MySQL. This allows us to the cache the results of queries to power reports, so that the data is available to everyone, whenever it is needed.

What next?

We’re looking for new ways to decrease our ingestion times from MySQL using stream processing with products such as Maxwell (http://maxwells-daemon.io/), which has been well-documented by Zendesk. Maxwell reads MySQL binlogs and writes row updates to Kafka as JSON. We’re also using SparkSQL and excited to use Apache Spark more broadly, especially Spark streaming.

We have a ton of enhancement requests to extend our Data Warehouse tables to meet the growing needs of the business and to provide better ways of visualizing the data via new Tableau dashboards.

As the Eventbrite family continues to grow with the acquisitions of Ticketscript, Ticketfly, and Ticketea, we continue to explore ways to migrate/combine data sources. This includes ingesting data from sources new to us, such as Amazon Redshift and Amazon Dynamo.

It’s fun times here at Eventbrite!

Special thanks to Eventbrite’s Data Engineering team: (Brandon Hamric, Alex Meyer, Will Gaggioli, Beck Cronin-Dixon, Jasper Groot, Jeremy Bakker, and Paul Edwards) for their contributions to this blog post. This team rocks!