Julio Zynger

Putting my inbox away

Mon, 20 Jul 2026 00:00:00 GMT

For about fifteen years my email "system" was: read a message, decide it was handled, and then leave it exactly where it was. I never archived anything. So my inbox quietly grew to 15,000+ messages, of which a grand total of twelve were unread. Every day I'd glance at whatever was new that morning and ignore the wall of history behind it. It worked, technically. It also felt like living in a room where nothing ever gets put away.

I'd been meaning to fix it forever. What finally did it was realizing I didn't have to do it by hand. A little command-line tool for Google Workspace had been rattling around my head ever since I stumbled onto Justin Poehnelt's post on rewriting CLIs for AI agents during some research for work. He built the thing to be driven by an AI in the first place, and it shows. I'd braced for a painful login-and-permissions ordeal to connect it to my personal Google account; it turned out to be basically one command. So I pointed Claude at it and, instead of clicking through thousands of emails, just asked questions: how big is this really? who emails me most? what are all these old labels?

The surprises were the fun part.

The first: my "unread" was secretly doing three jobs at once. It meant new, and I still need to deal with this, and I want to read this later, all at the same time. No wonder it never felt clean. Once I split those into separate places (using a Gmail setting called Multiple Inboxes I never knew existed until that afternoon), the whole thing calmed down. (There's a nice step-by-step walkthrough if you want to set one up yourself.)

Roughly how it looks now: only what's actionable stays in the inbox, things I owe a reply sit in "Follow up," and newsletters pile up in "To Read."

The second one I did not see coming. I went in expecting to find heaps of newsletter junk to unsubscribe from. There basically wasn't any. When I actually looked, the "spam-looking" senders were things like a job offer, a note from a recruiter, an apartment contract. My inbox was never clutter. It was an unindexed archive of pretty much everything that had ever happened to me, and I'd simply never asked it what was inside.

There was even something oddly sentimental in it. My old folder labels turned out to read like a timeline of my life: university, a year abroad, a startup that didn't make it, a job that led to another. I hadn't set out to keep a diary. It just accumulated.

By the end of the afternoon the whole pile was archived (not deleted, just put away and still searchable), the dead weight was gone, and new mail now sorts itself: newsletters wait in a "read later" pile, receipts file themselves out of sight, and only genuinely new things land in front of me. The one habit I kept is embarrassingly simple: when I'm done with something, I put it away.

It's a small thing. But my inbox finally feels like mine again, and I mostly just chatted with a computer to get there. 🤖

Why I stopped saying "legacy"

Thu, 02 Jul 2026 00:00:00 GMT

Outside of software, legacy is a word of pride. People talk about the legacy they want to leave — what they hope to be remembered for, what they want to pass on. The word is used with reverence.

In software, it's a complaint. The legacy database. The legacy frontend. The legacy auth flow.

I try to never use it that way.

In our industry, legacy carries baggage. It implies obsolescence, accumulated pain, something to be embarrassed about. And yet — in almost every case I've worked on — the "legacy" system is the one paying the bills. It's the system serving traffic right now, the one customers are happily using, the one funding the migration to its replacement.

As Bjarne Stroustrup observed:

Legacy code often differs from its suggested alternative by actually working and scaling.

The word shapes how teams treat it. Once a system is legacy, it stops getting investment. Bug reports get triaged differently. New hires are told not to learn it. Engineers who maintain it feel like they're working on something already dead.

And the label sticks. A system usually earns the "legacy" tag the moment a replacement starts being planned — but migrations drag, or don't finish at all. The negative connotation lingers in the organization for years. People want distance from it. Meanwhile the system everyone has decided is legacy keeps doing its job, waiting for a successor that may never quite arrive.

I use classic instead. The classic-cars analogy I owe to Kyle's post on migrations — a system that's earned its place, that we still respect, that we choose to maintain even as we build its successor. Vintage works too, with the same reframing.

This isn't just vocabulary. In the previous post, the renamed build script was proto:gen:classic, not proto:gen:legacy. That wasn't an accident.

When I find myself reaching for legacy, I stop. The system in front of me is probably what's keeping the lights on. I owe it a better word.

Default to the end-state

Wed, 17 Jun 2026 00:00:00 GMT

A platform team I work with is moving our compute from x86_64 to arm64. The migration plan, roughly:

Multi-arch the container builds.
Add graviton = true to the service's Terraform module call (touch every repository).
Once every service has flipped the flag, remove it (touch every repository again).

Step 3 is where I want to stop.

Step 3 is a second sweep across every repository that participated in step 2 — re-opening each service, deleting the flag, re-deploying. It is the most expensive part of the migration and the easiest to defer indefinitely. Most migrations never finish step 3.

There is a cheaper version of the same plan. Change the flag's polarity.

Instead of graviton = true, expose legacy_x86 = true. The default — the result of doing nothing — is now the migration target. Step 2 becomes "services that aren't ready opt out". Step 3 becomes "the platform team deletes the flag handling in a single PR". No second sweep. No long tail.

The same trap shows up outside of infrastructure. When we migrated our protobuf TypeScript generation from one toolchain to another, every package depended on a build script called proto:gen to produce its TypeScript bindings. The natural instinct was:

Introduce a second build script proto:gen:v2 next to the existing proto:gen.
Migrate every package to call proto:gen:v2 (touch every repository).
Once every package is on v2, delete the old script and rename proto:gen:v2 back to proto:gen (touch every repository again).

Same shape. Two sweeps.

The polarity-flipped version:

Rename the existing script to proto:gen:classic (touch every repository, but no behavior change).
Introduce the new generator under the canonical proto:gen name. New code lands on it by default.
Migrate packages off proto:gen:classic opportunistically, and delete it once nothing depends on it.

proto:gen always means the current canonical generator, so no package has to touch its build script just to keep up with the migration.

The underlying idea has a name borrowed from API design: the pit of success. Doing nothing should land you in the state you want. A flag — or a script name, or a default config — that points away from the target inverts that. Doing nothing keeps you in the past.

So when I find myself proposing a migration flag, I ask:

If every team ignores my Slack message, where do they end up?

If the answer is the old behavior, I flip the flag around.

Zenjob is Betting on TypeScript: An Organizational Decision

Wed, 27 May 2026 00:00:00 GMT

Adopting gRPC at Zenjob: The Technology Was the Easy Part

Tue, 31 Mar 2026 00:00:00 GMT

What Engineers Need from an All-Hands: It's Not Tech Updates

Tue, 27 Jan 2026 00:00:00 GMT

SoundCloud Echo: Next-Level Humane Registry with Backstage

Mon, 19 Sep 2022 00:00:00 GMT

Obvious Ownership: A Sensible Humane Registry

Thu, 06 Jan 2022 00:00:00 GMT

Did I Break You? Reverse Dependency Verification

Tue, 25 May 2021 00:00:00 GMT

Tests Under the Magnifying Lens

Tue, 02 Feb 2021 00:00:00 GMT

SQLite: when Insert means Delete

Tue, 30 Jun 2020 00:00:00 GMT

Modernizing some of SoundCloud's Android app storage layers, I've been especially invested in databases, and have been migrating a lot of the core entities of the app between our in-house ORM and industry-known alternatives, as Room and SQLDelight.

There are many parts to that work, ranging from understanding the current standings of the schemas to coming up with improvements, risk mitigation or reducing overall tech-debt. It has been a great opportunity to learn about the internals of SQLite, the RDBMS that powers most mobile apps, widely popular in Android but that also backs iOS' CoreData framework.

Today, I want to describe an interesting scenario we've hit while using a few more advanced features of SQLite: Foreign Key trigger actions, but more deeply, how their usage in combination to insertion statements can raise unexpected effects.

Foreign Key Actions

Let's imagine a schema with two tables, User and Track and a junction table TrackCreator whose schema is as follows:

CREATE TABLE IF NOT EXISTS TrackCreator (
  `track_id` TEXT NOT NULL,
  `user_id` TEXT NOT NULL,

  PRIMARY KEY(`track_id`, `user_id`),

  FOREIGN KEY(`track_id`) REFERENCES `Track`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE,
  FOREIGN KEY(`user_id`) REFERENCES `User`(`id`) ON UPDATE NO ACTION ON DELETE CASCADE
)

Having a junction table is great because it allows us to write queries that JOIN the columns of either of the referenced tables, for example, if we need to get the creator name for a given Track.

Now, if either the Track or the User that created it gets deleted or have its id altered, SQLite can automatically update our TrackCreator table if we set a Foreign Key action for these references.

SQLite's documentation does a great job explaining what foreign keys are, and more interestingly the possible actions one can register as triggers to an update or delete in the parent table.

Conflict resolution

SQLite also accounts for possible conflict resolutions when constraints are violated: on the example above, stating an INSERT for an already existing Track or User would violate the PRIMARY KEY uniqueness constraint on either table.

Most ORMs make it really easy to define a conflict resolution, altering the INSERT statement to include the ON CONFLICT clause:

When working with Room, that can be done on the Dao by specifying it in an annotation:

@Dao
interface TrackDao {
    @Insert(onConflict = OnConflictStrategy.REPLACE)
    fun insert(tracks: List<TrackEntity>)
}

The catch

Now, on the setup above, a surprising effect would happen when inserting an already existing Track or User. As SQLite's documentation states:

When a UNIQUE or PRIMARY KEY constraint violation occurs, the REPLACE algorithm deletes (…) prior to inserting or updating the current row.

So here we are:

And at that point, our ON DELETE action will trigger, effectively removing the entry from the junction table as well. That means queries that rely on the JOIN statement will yield no results.

Android specificities

During our investigation, we were made aware of this behaviour by reading SQLite log statements. On Android, the mechanism to enable logging isn't straightforward, but exists.

When inserting Tracks that violate the uniqueness constraint of the primary key, we confirm our hypothesis:

V/SQLiteStatements: /data/com.soundcloud/databases/sc.db: "BEGIN EXCLUSIVE;"
V/SQLiteStatements: /data/com.soundcloud/databases/sc.db: "INSERT OR REPLACE INTO `Users` (...) VALUES (...)"
V/SQLiteStatements: /data/com.soundcloud/databases/sc.db: "INSERT OR REPLACE INTO `Users` (...) VALUES (...)"
V/SQLiteStatements: /data/com.soundcloud/databases/sc.db: "-- TRIGGER room_table_modification_trigger_trackuser_DELETE"
V/SQLiteStatements: /data/com.soundcloud/databases/sc.db: "-- UPDATE room_table_modification_log SET invalidated = 1 WHERE table_id = 7 AND invalidated = 0"
V/SQLiteStatements: /data/com.soundcloud/databases/sc.db: "-- TRIGGER room_table_modification_trigger_trackuser_DELETE"
V/SQLiteStatements: /data/com.soundcloud/databases/sc.db: "-- UPDATE room_table_modification_log SET invalidated = 1 WHERE table_id = 7 AND invalidated = 0"
V/SQLiteStatements: /data/com.soundcloud/databases/sc.db: "-- TRIGGER room_table_modification_trigger_users_DELETE"
V/SQLiteStatements: /data/com.soundcloud/databases/sc.db: "-- UPDATE room_table_modification_log SET invalidated = 1 WHERE table_id = 0 AND invalidated = 0"
V/SQLiteStatements: /data/com.soundcloud/databases/sc.db: "-- TRIGGER room_table_modification_trigger_users_INSERT"
V/SQLiteStatements: /data/com.soundcloud/databases/sc.db: "-- UPDATE room_table_modification_log SET invalidated = 1 WHERE table_id = 0 AND invalidated = 0"

Starting on version 3.24.0 (2018–06–04), SQLite does provide an UPSERT statement (following the syntax established by PostgreSQL) but the bundled version of SQLite in the Android framework is not quite up-to-date.

It is still possible to 'manually implement' UPSERT by mimicking SQLite's syntax:

Time for a migration

In our case, we could relax our constraint definition by dropping the ON DELETE action trigger, instead of implementing the UPSERT. Here, the usage of SQLite brings a bit more work: the ALTER TABLE statements omit many of the advanced clauses other SQL engines provide, including CONSTRAINT manipulation related.

So, in order to change the constraint definition for the foreign key, we require a multi-step migration, that will copy the existing data into a temporary table and rename it afterwards:

database.execSQL("""
    CREATE TABLE IF NOT EXISTS TrackCreator_temp (
        `track_id` TEXT NOT NULL,
        `user_id` TEXT NOT NULL,
        PRIMARY KEY(`track_id`, `user_id`),
        FOREIGN KEY(`track_id`) REFERENCES `Track`(`id`) ON UPDATE NO ACTION ON DELETE NO ACTION ,
        FOREIGN KEY(`user_id`) REFERENCES `User`(`id`) ON UPDATE NO ACTION ON DELETE NO ACTION
        )
    """.trimIndent())

// Move all data to temporary table
database.execSQL("INSERT INTO TrackCreator_temp SELECT * FROM TrackCreator")

// Drop mis-scheme table
database.execSQL("DROP TABLE TrackCreator")

// Rename temporary table to definitive name
database.execSQL("ALTER TABLE TrackCreator_temp RENAME TO TrackCreator")

// Recreate indexes if needed (omitted for brevity)

Conclusion

One must be attentive to the underlying mechanisms of third-party tooling. Dealing with persistence and databases, it pays off to get acquainted to some degree of detail, especially given the cost of a post-factum change.

Being embedded, and in an almost feature-freeze on mobile OSs, SQLite's lean API still provides 'eureka moments'. There are always opportunities for learning.

FloorPlan: Visualize database evolution

Tue, 16 Jun 2020 00:00:00 GMT

Easier peer reviews and documentation-as-code

At SoundCloud, we are around 20 Android engineers, working in a multi-module project with more than 10 databases, with multiple tables each. And that's only the Android app!

As software matures, more functionality is added, code gets moved, re-written or removed. In the same rhythm, team members change and with them historical knowledge might get lost. For that reason, it is important to have a well-organized project, that is inviting for new contributors while still digestible for one-off readers.

Understanding how data is structured can be very helpful as an initial step for diving into the abstract modeling of the business domain. Database schemas, then, provide that entry window, but building the mental model out of a machine-readable format can be a daunting task even for seasoned engineers, especially when the project grows large and changes rapidly.

Enter FloorPlan

FloorPlan is an open source Kotlin library to translate database schemas into DBML definitions and ER diagrams.

It is distributed as a CLI tool and as a Gradle Plugin, for manual interaction and integration with CI environments.

floorPlan {
  schemaLocation = "schemas/"
  outputLocation = "floorplan-output/"
  outputFormat {
    svg {
      enabled = true
    }
  }
}

Check out the full documentation and sample plugin integration.

Versioning 🗒️

As you already do for the rest of your documentation, checking in human-readable representations of your application's logic is a great way to build historical knowledge on the codebase, and what ended up in production.

By using git, developers can iterate on proposed schema changes, easily review them and also bring in non-technical peers by lowering the learning curve into database technologies.

Code review 🔎

An useful recipe is to check-in each version of the database as a DBML schema, and then later, leverage the power of standard diffing tools to quickly spot changes:

$ diff -u 4.dbml 5.dbml
--- 4.dbml      2020-06-12 12:20:09.000000000 +0200
+++ 5.dbml      2020-06-12 12:33:15.000000000 +0200
@@ -1,9 +1,8 @@
 Table play_queue {
   _id int [pk, increment]
-  entity_id int [note: 'nullable']
-  entity_type int [note: 'nullable']
+  entity_urn varchar [note: 'not null']
   related_entity varchar [note: 'nullable']
-  source varchar [note: 'nullable']
+  source varchar [note: 'not null']
   context_type varchar [note: 'nullable']
   context_query varchar [note: 'nullable']
   played int [note: 'nullable']

In the spirit of full automation, these can then be written as pull request comments after a CI build:

Documentation 📝

With the power of GraphViz, FloorPlan can also output other rendering formats, such as vector SVGs, rasterized PNGs and diagram DOT files, allowing for even further integration with your team's workflow.

At SoundCloud, we host an internal documentation portal, completely based on markdown files, that now can automatically include diagrams for our client application databases as pull requests are merged:

Extensibility 🗜️

FloorPlan's API is designed for extensibility, and its Consumer API allows for integration with different RDBMS engines and ORMs. It doesn't matter which source database ORM schema structure you use, with the help of a Consumer implementation, FloorPlan will be able to translate the database schema.

On the Android space, for instance, there are many popular libraries to communicate with SQLite, each providing a different feature-set, that FloorPlan can integrate with.

That doesn't make its usage Android-specific, though, since a Consumer for MySQL, Postgres or Oracle could also be built by leveraging these RDBMS's schema exporting tools. In fact, one can work with already existing tools to use DBML itself as an input to FloorPlan.

Try it out

Integrate FloorPlan in your project's workflow, contribute and provide feedback so it can improve further!

Be a good client: request prioritization

Mon, 23 Mar 2020 00:00:00 GMT

The story is more common than it seems: a new feature is developed and tested extensively in-house, but only when the client applications are deployed, the engineering team discovers the extra production load and seasonal request rate. A hot-fix is needed! But now it's too late: we cannot force our users to update their apps or refresh the page on their web browser.

As we've discussed in part 1 of this series, clients are in a way bigger number than our servers, and even though we have applied some of the techniques previously described to make our clients smarter, there's more we can do on the server-side to make the entire system more resilient.

Remote Tuning

When communicating to a backend, it is a good idea for clients to expose as much information as possible about their requests. That way, a server will be in a better position to qualify and prioritize the calls it receives.

Keep in mind, the recommendations here apply only to technical aspects of the requests, and if you fully control the infrastructure — do not identify personal user information, and most importantly, do not share these with 3rd-parties.

Client identification

By exposing a client name and its version, the backend can categorize requests and even workaround possible client bugs. It is especially interesting to agree on a structured form for that value, allowing for parsing, sorting and filtering. There is already a standard for web browsers, but it proves very valuable on mobile apps as well.

func serve_request:
  if (client_id = android_app & client_version = 1234):
    // suppose there is a retry loop bug in that app version
    return Response(400)
  else:
    handle_request()

Another aspect that can be useful from a backend perspective is bucketing of client variability throughout a singular client version. For example, if there are feature toggles or A/B tests in place, sharing variant identifiers will be helpful to identify and isolate arising issues.

Serving order

In the event of a partial outage, servers can decide to drop requests to relieve load in the overall system. In fact, the faster a circuit-breaking mechanism triggers, the more resources will be saved. API gateways are good candidates for such logic, but in some cases it can also be applied throughout the service net.

Introducing a company-agreed index of criticality for certain features and severity levels in case of degradation will support ranking which paths have serving priority. The server can then reject requests that fall under the lowest levels of the ranking. As exemplified by Google's SRE book:

For example, when a system displays search results or suggestions while the user is typing a search query, the underlying requests are highly sheddable (if the system is overloaded, it's acceptable to not display these results).

The criticality index can be composed by several sub-metrics: for example, if a serving-path goes through usually overloaded services or of expensive scalability or if it is known to have caused retry storms in the past.

We can also take in account UX-aspects to bump the criticality of a request path, for example, whether a request was triggered by a user or happened from an automated job action, or whether the client application was in foreground or background in a mobile device.

As an alternative, besides rejecting requests, one can also decide to 'gracefully degrade' the response, skipping the expensive paths that aren't critical to the overall user experience. For example, suppose fetching an audio track metadata from a saturated backend, a server can decide to skip calculating how many likes the track has and use a sensible default instead:

Budgets

Additionally to circuit-breaking limits for retries per request or per route, a client can also choose to apply a retry budget to limit the incoming load on the server and prevent request storms. Libraries like Twitter's Finagle and Linkerd bundle in that concept and allow for customization of the budgeting rules.

In general, that will mean that the application will keep track of the ratio between incoming requests and retries, and define a threshold as a configurable limit, that applies over a limited period of time.

Once a client goes over the threshold for the stipulated time-frame, it has its requests cancelled. In other words, a client can retry as much as it want to, as long as the ratio is maintained.

Since large applications tend to be a stack of services with dependencies on each other, it is key that requests are only retried at the layer immediately above of the rejecting one. On the example above, a rejection on the first service would prevent clients (pictured by the phone icons in the image) to retry further and save all potentially incoming combinatorial load on the database server down the line.

A more generic perspective over that concept is to establish a concurrency limit, which will bound the total number of requests a server will handle concurrently, and optionally provide a queue of waiting requests. All of the incoming requests on top of these values get automatically rejected.

What we've seen throughout this small series is that resilience is a characteristic of a system as whole, and it builds on top of all of its individual pieces.

On part 1 we have optimized the requests' path from the clients' perspective, on part 2 improved the relationship between client and servers while still providing a good user experience and here on part 3, we got back control from the field by moving toggles, limits and flags to the backend.

In summary, we've collected a few techniques that will help scaling products in a way that we can also rely on them even in the event of a full or partial outage scenarios.

Be a good client: retries

Wed, 11 Mar 2020 00:00:00 GMT

When a request to a server fails, it is very tempting to issue another try to the same route, as a way to increase reliability of systems or mask transient problems to the end-users.

How often is this a good idea? Never. Well, more or less.

Today, many RPC libraries provide developers with built-in functionality for retrying requests, making it really easy to hammer servers with more and more load. Thus, retrying requests by default is not a good idea, instead, one must consider the situation and only then decide whether a retry is worth it.

More jitter

In the event of a transient outage, or full unavailability, retrying clients will increase the load on the server, potentially causing an overload in what would otherwise be a regular number of requests, causing even more clients to issue retries. This snowball effect is what is effectively alleviated by introducing jitter into the system, not only before the initial request, but also in subsequent retries.

Exponential backoff

There are several techniques that an engineer can employ to further relieve stress on their servers, of them, exponential backoff is the most popular. In a nutshell, this means varying the amount of time between retries instead of having a fixed time window.

func sync() {
  val jitter = random(0.5, 1.5)
  val back_off_wait = 0.5 // waiting time between requests
  wait(jitter.to_seconds)

  success = perform_network_request()

  while !success {
    wait((back_off_wait * jitter).to_seconds)
    success = perform_network_request()
    back_off_wait *= 2 // increase waiting time per attempt
  }
}

Circuit-breaking

Making the waiting time longer would help even out the load in the event of a transient server outage, but very rapidly this pseudocode above would cause clients to wait an unreasonable amount of time to then perform their follow-up request.

Not only this is a bad idea from a UX perspective, it would also mean that all of this load would be then deferred to a later point in time in which the server would already be correctly functioning, and possibly the user isn't interested in its outcome anymore.

So, besides backing-off, one would also define a circuit-breaking mechanism to prevent useless or unwanted retries. In this case, a maximum threshold for the waiting time between retries before informing the end-user of the failure.

func sync() {
  val jitter = random(0.5, 1.5)
  val back_off_wait = 0.5
  val max_waiting_time = 10 // circuit-breaker
  wait(jitter.to_seconds)

  success = perform_network_request()

  while !success && back_off_wait <= max_waiting_time {
    wait((back_off_wait * jitter).to_seconds)
    success = perform_network_request()
    back_off_wait *= 2
  }
}

Error codes and HTTP

There are cases in which a client can be assured a server will not be capable of fulfilling a request, which we can account for. In fact, HTTP defines codes that one can use to help decide whether to issue a retry, becoming part of the circuit-breaking condition.

Clearly, one should not retry requests that failed because of client errors (for example, a missing query parameter, or incorrect Authorization header). Independently of the load on the server, these requests would result in the same errors if retried.

It is, however, potentially a good idea to retry both network errors and server errors. The first could happen for a temporary fault in connectivity and possibly recovered later on, while the latter could ensure recuperation when the transient outage is resolved or the load is relieved.

The server in command

For a few specific status codes, headers can also be used to pass additional information to optimize communication timing and reduce load. In the event of rate limiting (429), a known interruption of service or overload (503), the Retry-After header can be used to indicate how long the client should wait before making another request.

Seen as a hint, it is still responsibility of the client application to change its behavior given the presence of that header. In fact, many modern browsers already support its detection, and even Google's crawler is aware of it to determine when to visit websites.

This mechanism is especially interesting since having a dynamic value means moving the control of the request streams to the server, allowing for it to adjust accordingly to its current incoming load. In fact, all of the techniques mentioned above could also be applied by having remotely-fetched values, from the threshold of retries to the amount of jitter per request. The client, then, becomes responsible only to define sensible defaults.

When dealing with I/O and networking, the one certain thing for every engineer is that there must be error handling logic in place. Even better, when possible, we can pretend errors never happened by gracefully recovering! Being conscious of how clients and servers talk, and keeping in mind the trade-offs within our infrastructure, the smarter we try, the better.

There's even more we can do to make our client-server relationship a healthy one. So far, we have looked into techniques for full outage scenarios, but being a good citizen also means behaving well when partial failures are ongoing or ideally to prevent them altogether. We'll dive deeper in these in part 3 of this series.

Be a good client: jitter

Wed, 04 Mar 2020 00:00:00 GMT

Periodically triggered jobs are very common in modern applications, especially in the client-server model: uploading notes, analytics events or backing up data are good examples of timely operations that a developer might decide to trigger at a specific rate, for example once per hour, or at 3 a.m. when the user isn't actively interacting with the device.

Now, handling the computational load can get tricky since the number of clients is a magnitude larger than servers.

It turns out that, as your application gets more successful, more clients will synchronize requests to your backend. That means your servers will see higher network traffic as these timely operations trigger (in a similar fashion to what's known as the Thundering Herd problem), as such:

For your users, the higher load in traffic will mean higher response latencies, and potentially errors due to network timeouts. In a catastrophic scenario, a big enough number of clients or scheduling of call retries could even lead to denial of service (DDoS): what would be a small outage or a quick interruption of service will snowball by having clients enqueueing more and more retries that would themselves also fail.

Operationally, this would mean a waste of resources and will translate as an increase in cost to maintain your infrastructure. While most of the time the infrastructure will be idle, engineers will over-scale the services to be able to serve the high load of requests on the clock-tick moment.

Ideally, we want to distribute the load evenly across time, while still being able to serve the same number of requests, or in a visual way, this is how the distribution should look like:

We can do so by purposefully introducing jitter:

Jitter is the variation in periodicity of a signal or periodic event from its target or true frequency.

In practical terms, this means purposefully advancing/delaying the next call outside of what the expected clock tick would be. For a network request that usually would take an order of hundreds of milliseconds to be performed, we can introduce a delay of a few milliseconds to even out the load on the servers.

func sync() {
  // instead of immediately performing the call
  // we add jitter to distribute the network load
  val jitter = random(0.5, 1.5)
  wait(jitter.to_seconds)
  perform_network_request()
}

Some libraries feature configurable jittering (see Netflix Hystrix, Resilience4J, Twitter's Finagle, Google's HttpClient, Android's WorkManager), while others give full control (and responsibility) to the application developer (.NET Polly, OkHttp, NSURLConnection).

The key here is understanding the specific use-cases, and where the sweet spot is to balance incurring traffic in your server infrastructure and data consistency between client and server.

Watch on YouTube

Jitter is especially important when associated to other resiliency techniques, like batching, call retries and exponential backoff. We'll look further at these in part 2 of this series.

Media Projection and Audio Capture

Wed, 30 Oct 2019 00:00:00 GMT

Starting from Android Lollipop, developers have an API that can be used to capture parts or the entire visualization of a device's screen: MediaProjection. James O'Brien gave a great description of that usage in this article.

From Android 10, the MediaProjection API was extended to support the audio capture use-case. That is especially interesting if your app does any sort of streaming or Twitch-like broadcasting. In some cases, though, you might want to have finer control over when or what can be captured, either for user-privacy reasons, or content-protection (i.e copyright).

At SoundCloud, we cared for both use-cases; and as we worked on preparing our app for targeting API Level 29, making sure only the expected actors could capture our audio was critical. We want to be able to support use-cases like Live-Caption, but in turn, also protect our copyrighted content to prevent leaks or piracy.

To do so, simply following the documentation wasn't enough; we also wanted to verify the solution actually worked. For that reason, we have built a demo application that uses the audio capturing API to interact with our app, just like a third-party app would do. Here's how we've done it.

Before we get started

Given we will be recording what the user's device is playing, we are first required to request a few permissions. Make sure to prompt the user at the appropriate time for the RECORD_AUDIO permission.

Also, since the audio capturing operation will be long-standing, we will need a foreground service to keep the user informed of the execution. For that reason, don't forget to declare the FOREGROUND_SERVICE permission on your AndroidManifest file, too.

<manifest>
  ...
  <uses-permission android:name="android.permission.RECORD_AUDIO />
  <uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
  ...
</manifest>

Audio Capture

For privacy reasons, audio/video capturing is special on Android in comparison to other permission requests, in the sense that a capturing app must prompt the user for explicit approval every time a MediaProjection is needed.

A request to the MediaProjectionManager system service can be done with a single-liner, but must happen on the context of the UI so that a confirmation dialog can be displayed.

From Android 10 and later, a Service must be running and call startForeground to post a Notification before we can obtain the MediaProjection instance; failing to do so will cause a SecurityException. The actual audio capturing operation does not need to be done in the Service code, but you absolutely need a Service, even if its sole purpose is to manage the lifecycle of the Notification. Don't forget to declare the foregroundServiceType on your AndroidManifest declaration:

<manifest ...>
  ...
  <application >
  <service
      android:name=".AudioCaptureService"
      android:foregroundServiceType="mediaProjection" />
    ...
  </application>
</manifest>

Now that we have all the pieces in place, we can obtain MediaProjection instance to perform the audio capturing.

Audio Capture

Here is when things get interesting; the audio capture configuration API is very flexible and provides many hooks for its definition, so we can optimize our specific media use-cases. We will pass both an AudioPlaybackCaptureConfiguration and an AudioFormat object to the AudioRecord instance we will use to fill our audio data buffers.

The first object will define which type of media we will capture (USAGE_MEDIA, USAGE_GAME), and we can optionally define inclusion/exclusion app UIDs to filter which apps we are (or aren't, respectively) interested in capturing.

The AudioFormat defines how the audio data will be encoded. We can set the capture sample rate in Hz, the number of channels for capture, and which encoding to be used; from raw PCM samples with values varying from 8, 16 or floating point precision, to even compressed samples of different types, varying from MP3, AAC, AC3 and more, based on the devices encoding capabilities.

Notice how important these parameters are depending on the use-case: if you plan to upload the capture audio to the cloud or store them to disk, you might prefer encoded samples for their lower sample/byte ratio, but on the other hand if your use-case is of demuxing or post-processing you might be interested in more precise renditions in PCM.

For our example, validating our first-party app being captured by a third-party, it is enough to define the simplest combination of static properties: mono PCM-16 raw audio. For a more complex use-case, one could make them dynamic based on the target captured material properties. Fortunately, the documentation for AudioFormat is quite extensive and describes well all available usage options.

val config = AudioPlaybackCaptureConfiguration.Builder(mediaProjection)
    .addMatchingUsage(AudioAttributes.USAGE_MEDIA)
    .build()

val audioFormat = AudioFormat.Builder()
    .setEncoding(AudioFormat.ENCODING_PCM_16BIT)
    .setSampleRate(8000)
    .setChannelMask(AudioFormat.CHANNEL_IN_MONO)
    .build()

audioRecord = AudioRecord.Builder()
    .setAudioFormat(audioFormat)
    .setAudioPlaybackCaptureConfig(config)
    .build()

Holding the AudioRecord object, we can call startRecording to initiate the flushing of audio samples into our predefined buffer, which can be an in-memory or file OutputStream.

Notice this is a performance-critical operation: interruptions on the read thread will cause audio glitches and crackling (read more about low-latency audio rendering on my other article). On top of that, we also don't want to block the UI thread with our recording execution, so it is advisable to run the recording in a Thread of its own.

In our example, we will convert the PCM-16 integer samples into a ByteArray to be written to disk, so we must keep note of the endianness to be able to properly perform the samples' playback later on.

Once we're done with the capture, we stop the AudioRecord, release all heavy resources and stop our foreground service.

Once with the data…

Performing playback of the captured PCM data on Android is possible, even though it isn't done with the friendlier or more commonly used APIs of MediaPlayer, but by using AudioTrack.

For that reason, and for visualization purposes, we suggest pulling out the captured data for processing with a desktop-app solution such as the free Audacity audio editor. There, we have fine-tuned control to import the raw data and specify all of the parameters we have previously defined for encoding, sample rate, bit precision and even byte endianness.

By using the demo app, we could then verify that capturing of audio samples only happened for the media we explicitly specified, and all of our content was protected according to our business requirements.

For more details of how the API is designed and insights to our experiment at SoundCloud, check out the public GitHub repository for the sample app we've built. You can use it to verify your app reacts as you expect to audio capturing. There, you will also find a full implementation, deeper clarification and code-comments for further adaptation of the code to apply it to your recording use-case.

julioz/AudioCaptureSample - Sample app for Android 10's AudioPlaybackCapture API, which allows applications to capture the audio of other apps.

Thanks to Riccardo and Danny for the proofreads.

SoundCloud Is Playing the Oboe

Fri, 21 Jun 2019 00:00:00 GMT

Native Code and Debug Symbols

Tue, 23 Apr 2019 00:00:00 GMT

It is a common misconception to think that software development is all about writing code, but in reality, that is not where engineers invest most of their time. Instead, most of the effort is put in designing the application and understanding how it behaves once it is deployed.

As applications grow larger and more complex, it is critical to establish a process around how bug fixing can be organised. For Android apps, there are multiple tools that will support teams to collect and analyse crashes and other debugging information.

Similarly to applications that include JVM code (Java/Kotlin) shrinked with tools like Proguard, if the application includes native code (usually C or C++), it might be trickier to get actionable information from stack-traces/tombstones.

Debug symbols

When a native binary is compiled into a shared object (.so), it might include debug symbols, which represent additional information on top of the symbol table of the program. In other words, they include information such as function and variable names, and many other bits related to the original source code.

View embedded content

Shameless plug: Here, I cover how native libraries get linked and loaded in a program to be run on Android

All this extra data can take up a lot of space (sometimes multiple times the size of the actual object code!), and especially for Android, that might inflate the final APK size in a considerable amount, since devices of different CPU architectures also require matching shared object files. For that reason, build tools (such as Gradle) decide for removing the extra debug symbol table, in a process called stripping (read more about strip).

We can use tools like nm to display the symbol tables of binaries. For example, when I run it over stripped and symbolicated builds of the same library, the output is:

$ nm flipper-stripped/obj/armeabi-v7a/libflipper_stripped.so
>
$ nm flipper-stripped/obj/armeabi-v7a/libflipper_stripped.so | wc -l
> 0
-----------------------------------------------------
$ nm flipper-symbolicated/obj/armeabi-v7a/libflipper_symbolicated.so
> ...
> 002b2540 W _ZNK11M3UPlaylist8toStringEv
> 00251abc W _ZNK12JniException4whatEv
> 002aa5ec W _ZNK12PropertySets3HLScvN4Json5ValueEEv
> 002ab008 W _ZNK12PropertySets4HTTPcvN4Json5ValueEEv
> 00336324 T _ZNK13SQLiteBackend15getRowStatementEv
> 0033766c W _ZNK13SQLiteBackend16hasBuiltInCryptoEv
> ...
$ nm flipper-symbolicated/obj/armeabi-v7a/libflipper_symbolicated.so | wc -l
> 407230

Not surprisingly, when building an AAR that contains each of these builds, the disk space usage difference is noticeable:

$ ls -lh
> juliozynger  26M Apr 21 flipper-stripped.aar
> juliozynger 113M Apr 21 flipper-symbolicated.aar

As you can see above, while stripping binaries is a great idea for saving space, it will prove challenging when an engineer needs to debug an application. So, similarly to how Proguard mapping files are exposed during the build process, native debug symbols can also be stored separately from the release binary and later used to symbolise it with tools like ndk-stack or addr2line.

Usually, that means the release binaries are stored in the jni subdirectory and the symbolicated binaries in the obj subdirectory. Some third-party crash-reporting SDKs, like Firebase Crashlytics, BugSnag, etc. take advantage of that standard and provide ways to upload the debug symbols so that an engineer can get digestible information directly from their web dashboards.

If you build your own native libraries, make sure to output the binaries following the directory arrangement (jni→obj) to benefit from the automatic upload. If you have a more complex setup for building your custom library (for example, stripping symbols in a later stage) and use Gradle, you can also disable stripping of symbols by adding the following to your library's AAR generation script:

apply plugin: 'com.android.library'

android {
  ...
  packagingOptions {
    // specify the path to your object binaries, or generally:
    doNotStrip '**.so'
  }
}

Here you can find a simplified version of the script we use at SoundCloud to make sure we store our symbolised and stripped binaries in the correct directories.

Attributions:

Icon: collecting by Takao Umehara from the Noun Project
Icon: QoS by Stefan Traistaru from the Noun Project
Icon: Bug fixing by Symbolon from the Noun Project
Image: Fabric's symbolicated crash

Once again, many thanks to Miloš Pešić for reviewing the content

Release Quality and Mobile Trains

Wed, 03 Apr 2019 00:00:00 GMT

Your app and low-latency audio output

Mon, 18 Jun 2018 00:00:00 GMT

When building any software, we want to provide the best possible experience to our users, giving them the feeling of being in full control and taking the most value out of our applications. With audio-related software, that is not different: a very important metric for tracking playback performance is latency: minimizing the time it takes from pressing the 'play' button and listening to music is key for providing a seamless impression.

Today we will visit some concepts on digital audio, learn about how part of it is done on Android devices, and understand how buffers play a key-role in providing the smoothest playback experience to users as possible.

Sound travels in waves…

…and a very common method to represent analog signals (such as waves!) is pulse-code modulation (PCM). Basically, the wave amplitude is measured at a regular time interval. Each one of these values is called a sample. A lot of these samples are necessary to represent actual sound — for example, more than 44 thousand measurements are made every second!

Stereo sound is nothing more than two waves being transmitted at the same time (nowadays, surround systems provide even more than two channels!). If we want to play stereo sound, then, we'll need to sample the two channels and digitally store them. In PCM terms, a frame is a set of one sample per channel. So, for stereo sound, a PCM frame will contain data representing two samples.

Every modern phone out there has, as part of its circuits, a component capable of converting digital data (0s and 1s) to analog data (in this case, variable voltage, that will be passed to speakers). These are called digital-to-analog-converters (DACs).

To process data, they require information to be fed at a constant rate, and also in blocks of a specific size. The DAC knows where it should look for data — that is a region of memory the operating system specifies and in which we (application developers) should write to, with a very fancy name: the buffer.

The DAC will reach out to the buffer at a fixed time interval to get one of these data blocks (or chunks), but if we fail to write to the buffer when the DAC visits it, the hardware will output silence. That is a very bad experience for users and the reason music stops playing or they listen to crackles/noise (a.k.a. popcorn), so we should avoid it at all costs, by constantly writing to the buffer as soon as possible.

This is where latency comes in — the time we take to feed the DAC (or, in other words, to fill the buffer). In Android, audio tasks are among the highest priority ones in the entire system: a completion of one of these tasks will even interrupt other unrelated tasks in the OS level, so that the processing is as fast as it can be.

A special characteristic of Android devices…

… is that they can come in many shapes and sizes, but also hardware architectures, and manufacturers ship different DACs within their boards. To be able to get the most (or the least!) out of the latencies, we can also 'ask' the OS (Android 4.1 and above) what is the most suitable configuration for their buffer so we can work the best with the embedded DAC.

By metrifying SoundCloud's app, we were able to collect some interesting statistics about how variable were the hardware configurations for this considerably big userbase (taken in mid-2018):

Sample Rate (Hz)	Ratio
48,000	87%
44,100	12%
8,000	< 1%
96,000	< 1%
88,200	< 1%
32,000	< 1%

Buffer size (#frames)	Ratio
192	30%
240	27%
960	22%
1024	9%
256	3%
480	2%
2048	2%
96	1%
512	1%
1920	1%

Sample rate

One of the things we can do is to configure our application to fill the buffer with the optimal number of samples per second — the sample rate. Usual examples of sample rates are 44.1kHZ (i.e: 44.100 samples per second) or 48kHZ, but different DACs might have different optimal values.

If you don't use the DAC's expected sample rate, the OS will have to resample your data to match that, and then the audio processing won't go through the fast path (taking precious milliseconds from our low latency goal).

We can use one AudioManager API to query that value from the system and then pass it to the player in our application:

val audioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager
val outputSampleRate = audioManager.getProperty(AudioManager.PROPERTY_OUTPUT_SAMPLE_RATE).toLong()

Buffer size

If we use a buffer that is too small and the CPU has a lot of tasks to perform, there could be not enough time to come back to our thread and fill the next chunk of the buffer, so we'd get popcorn, because the DAC would have to wait for the next chunk to be delivered.

On the other hand, if we use a buffer that is too large, that means we need to delay the output of audio for the time duration of that buffer while we fill it completely before we deliver it to the DAC — and there is latency again. For example, if we have a buffer that can hold 4096 samples and our sample rate is 44.1kHZ, that means we would have a ~93 ms delay from when the data comes in the processing pipe and when the DAC is able to consume it.

Yet again, we can ask the OS what is the best buffer size we should be using, to find what the optimal value is:

val audioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager
val framesPerBuffer = audioManager.getProperty(AudioManager.PROPERTY_OUTPUT_FRAMES_PER_BUFFER).toLong()

We will want then, to use a multiple of that value to determine how many samples we should put in each of the buffer's chunks, so that we can minimize the number of times the DAC needs to call us back for more data.

For example, if the device reports a recommended buffer size of 208 samples and we make our chunks hold a non-multiple value of samples, like 160, we could have the following scenario:

Let's say the first 160-samples chunk was consumed, and the DAC calls us before the buffer-filling thread is able to add more data. Since there is not enough data in the buffer to collect another chunk, the DAC will have to call us yet again later. In the meantime, the OS scheduler might interrupt the DAC thread to give CPU access to another task. If that's the case, we run the risk of surpassing the time deadline to output audio, causing a glitch.

Interruption of DAC's consumer thread is more likely than we would expect, given that the platform uses locks around the buffer data structure, preventing concurrent reads and writes. So even if the DAC gets scheduled at the right time, but the filler thread is interrupted, it will have to wait for the lock to be released — surpassing the time deadline (i.e: glitch). It is something Google is aware, though: they recently released a new audio API for usage with Android Oreo and above that claims to improve that scenario.

In summary,

there are many factors that influence latency and the overall feeling we give users when doing audio playback, and some of these are even out of our control.

For the aspects we do control, it is beneficial to have a broad perspective of the elements in play, and learning how the underlying operating system behaves when we interact with it is certainly very helpful.

Resources:

Extending the Web with Android Instant Apps

Thu, 18 May 2017 00:00:00 GMT

During I/O 2016, Google announced it was developing a new way to bridge the gap between the web and native apps, by making them as easy to access as a simple click in a link: Instant Apps, made of small data-chunks that can be temporarily loaded on the device with no extra going-to-the-Play-Store steps.

Selected developers started adapting their apps to the new format shortly after: the feature rollout will happen in stages, beginning a small set of devices and starting on Android Marshmallow and above.

In this post, I'll introduce a little of my experience with the Instant Apps SDK and provide some insights while tinkering with the code. Hopefully this will be helpful in migrating your existing application.

Watch on YouTube

Preparing the project

Earlier this year, Google released a list of steps to help you prepare your app to supporting Instant App experiences. The initial steps should be already covered by your application at this moment: support Android Marshmallow's runtime permissions and deep links to enable URL-based navigation (Android Studio can help with that!).

Take into account that if your app requires logging in, you'll also have to integrate the SmartLock API for Passwords. Also you can use the new Payments API to a include checkout flow (Google currently has partnerships with Vantiv, Braintree and Stripe, but more to come).

The most complicated step might be modularizing your application as each downloaded module on a cold start should have less than 4MB, both for business (users on a mobile network might not be able to download bigger files) and technical reasons (Google's CDN is optimized to distributing files up to 4MB in size).

You can also have 2 versions of your app: Full App & Instant App. However, this is rather discouraged by Google. Ideally your instant app should have a subset of features of your full app.

Features or APK splits

The Instant Apps SDK introduced the concept of APK splits (also called features - not to be confused with this kind of split), that represents each module of your application. Android is able to download each of these features based on the accessed deep link — as long as there is at least one activity in the application that handles the URL.

Those modules are kept in a device-wide shared LRU-cache for future usage. Eventually, if the user accesses an URL that is not present in-cache, then the new download gets triggered.

If your app contains more than one single feature, then those will be bundled in a zip file also known as APK Bundle, which basically wraps everything for easier publishing.

Notice the size limit of 4 MB applies to the entire downloaded feature, including its dependencies. So if your base module (let's say, general-usage libraries and resources) already accounts for 3.5 MB, you're going to have a bad time delivering your app. Make sure your modules are lean: you can use one of many tools to help you out in this task, like ApkAnalyzer.

Building your first feature

In reality, features are just "pointers to libraries with a naming convention for unique identification". Android infers the dependency graph as shown in the below example:

Notice how a feature might depend on one or more libraries, and the libraries themselves might depend on other libraries (#mindblown). All features of the application must depend on the base feature, as that is the module that will contain the wiring of the instant app and also the definition of the deep links' URLs (more about that in the next section).

For each of your features, you'll need to add a module to your project, containing just a manifest with your Activity declarations and a build.gradle file that applies the com.android.feature plugin (or you can do it via Android Studio's wizards):

apply plugin: 'com.android.feature'

android {
  ...
}

<?xml version="1.0" encoding="utf-8"?>
<manifest package="com.sample.instantsonar.artists"
  xmlns:android="http://schemas.android.com/apk/res/android">

  <application>
    <activity
      android:name=".ArtistsActivity"
      android:theme="@style/AppTheme">

      <intent-filter>
        <action android:name="android.intent.action.MAIN" />
        <category android:name="android.intent.category.LAUNCHER" />
      </intent-filter>

      <intent-filter
        android:autoVerify="true"
        android:order="1">
        <action android:name="android.intent.action.VIEW" />

        <category android:name="android.intent.category.BROWSABLE" />
        <category android:name="android.intent.category.DEFAULT" />

        <data android:host="instantsonar.com" />
        <data android:pathPrefix="/artist" />
        <data android:scheme="https" />
        <data android:scheme="http" />
      </intent-filter>

      <meta-data
        android:name="default-url"
        android:value="https://instantsonar.com/artist" />
    </activity>
  </application>
</manifest>

Notice the IntentFilter I have declared to the activity. This is the definition of the deep links that will be handled by the Instant App: the feature that contains ArtistsActivity will be downloaded when https://instantsonar.com/artist (also for http) is accessed (as declared with the pathPrefix attribute, so it won't conflict with the other declared links).

In case your app declares more than one activity to match a given URL, the android:order attribute is the one that will define which will handle it (that parameter is optional and defaults to zero): for example, if you have an activity that declares the URL path /artist and another with /artist/*, when receiving a request from https://instantsonar.com/artist/42, Google Play will pick the activity with the lowest android:order value.

Another important aspect of the manifest are the <metadata> tags defining Default URLs. For each CATEGORY_LAUNCHER / ACTION_MAIN activity, you need to define a sibling default-url tag that will enable Play Store discovery and integration with Launchers.

Base feature

Your base library contains all the code necessary to run the core functionality of your application, so this is the place where you'd set the shared resources/assets and dependencies on libraries (as the support library and your preferred libraries for networking, dependency injection, analytics, crash reporting, testing, etc). All your features must depend on this base one: always take in account the size of those dependencies to not bloat your instant app's modules!

It is also the place where the declaration of your full application will reside, thus propagating the applicationId to all features in the project.

In order to differentiate the base feature from the others, all you need to do is to set a gradle attribute. The base feature build.gradle file will look something like:

apply plugin: 'com.android.feature

android {
    ...

    baseFeature true
  }

  feature project(':feature1')
  feature project(':feature2')
}

And its manifest file will be very simple, just containing permission declarations and shared activities (if that's the case for your project). Later on, the manifest merger tool will also aggregate all your module's manifest files:

<manifest package="com.sample.instantsonar.base"
    xmlns:android="http://schemas.android.com/apk/res/android">

    <uses-permission android:name="android.permission.INTERNET" />

    <application
        android:allowBackup="true"
        android:label="@string/app_name"
        android:name="com.sample.instantsonar.SampleApplication">

        <!-- shared activities here (only if you need!) -->
    </application>

</manifest>

APK Bundle

Building the APK Bundle is the simplest part of the process: all you need is to create a new module that contains a gradle file that will instruct how your bundle will be assembled.

Make sure you declare com.android.instantapp as the build plugin, and refer to all your features in the dependencies block.

Generally, the file will have a similar structure as this one:

apply plugin: 'com.android.instantapp'

dependencies {
  implementation project(':base')
  implementation project(':feature1')
  implementation project(':feature2')
}

This article was written during the early days of Android Instant Apps. The technology has since evolved into Android App Bundles and Dynamic Feature Modules.

Continuous Deployment no Android: como usar a Publishing API para automatizar seu release

Tue, 10 May 2016 00:00:00 GMT