Performance benchmarks¶

This page is trying to go through some performance metrics and see how better/worse Datafaker is in compare with Java Faker and other similar projects.

Hardware & Software¶

All the tests are done with help of JMH at Fedora 36 for different JDKs (mentioned in tables below). The laptop has 32Gb of RAM, AMD Ryzen 7 PRO 5850U with Radeon Graphics.

Datafaker (1.4.0) vs Java Faker (1.0.2) vs Kotlin-faker (1.11.0) vs JFairy (0.6.5)¶

Originally Datafaker started as a fork of java-faker. For java-faker there have been mentions of the poor performance of the library, and this document will compare the performance of the respective libraries against each other.

Recently, the 1.4.0 version of Datafaker has been released, and we'll look into what this says about performance improvements.

To demonstrate the performance, we're going to use JMH benchmarks. All the code is available as a separate project because of JMH license reasons.

Since it's hardly possible and reasonable to compare each method's performance, we'll focus on a subset of features in this article. In the below, you can see that Datafaker is about 10x-100x times faster for several cases.

JDK effects¶

Also, there is an interesting fact: in case of Datafaker moving from jdk8 to jdk18 performance is improving up to 25%, while for Java Faker it is not happening. It seems in Java Faker there are some time-consuming operations blocking such improvements from updating java version which partially were fixed in Datafaker.

There is also Kotlin-faker, which a performance comparison table in README page.

Unfortunately there is not so much information what kind of benchmarks have been executed. After some search across the project I was able to find only this page So, it looks like the only test which were done is checking Faker.name().name() performance.

Ok, let's start with the similar test here. Similar, because we are going to use JMH which was not used in their test. If applicable we try to execute same tests we did for previous section. So let's start with the original test from Kotlin Faker

Original Kotlin Faker Test¶

(for different libs there should be different classes, for more details look in the code): net.datafaker.benchmark.kotlinfakerbenchmark

ATTENTION! In this test we measure not throughput but the average time of one operation => lower is better.

In general, we could say that for jdk8 and jdk11 timings for Datafaker and Kotlin Faker are more or less similar and for jdk18 Datafaker is about 20% faster. JFairy and Java Faker are far behind.

Initialization¶

It's worth to measure since initially during initialization of Faker object it requires to initialise all the providers objects and read all the yaml files for providers.

Tests for initialization could be found at net.datafaker.benchmark.initialization Initialization:

Simple methods fullname, firstname, address¶

Performance of simple method calls like fullname, firstname, address.

Tests could be found at net.datafaker.benchmark.simplemethods

Firstname:¶

Fullname:¶

StreetAddress:¶

String template operations¶

From one side Kotlin Faker and JFairy do not support expressions, from the other side Kotlin Faker supports numerify/bothify/letterify and regexify operations.

So, the tests are done based on Kotlin Faker's doc page

Numerify:¶

Letterify:¶

Bothify:¶

Regexify:¶

Expressions¶

Since both Java Faker and Datafaker provide expression functionality like letterify, bothify, regexify, it would make sense to see the difference. Here, it is also worth mentioning that for different patterns score could be different, however it is impossible to check all.

net.datafaker.benchmark.templatestrings.DatafakerTemplateStrings

Bothify expression:¶

Letterify expression:¶

Regexify expression:¶

Method invocations¶

Also, both Java Faker and Datafaker allow invocation of methods from expression. Let's consider an example where in expression it is required to parse and process only one method. In Datafaker, a was added cache for the parsing of methods. Since only Datafaker and Java Faker support method invocations, there are only these 2 projects tested:

Similar example as previous, however there are 3 methods. Besides, cache of parsed methods there was also added cache for parsed args.

It makes sense to keep in mind that these tests do not cover all possible use cases and could be considered only as a starting point for analysis.

More Fun¶

There is an issue in Java Faker about generation of 100M of objects. Of course, the task could be solved with concurrent generation in multiple threads. However, here it is interesting how much a 1 thread application can generated in 1 hour.

The code below for Datafaker generates a bit more than 170M objects for 1 hour. The same code for Java Faker generates about 38M for 1 hour, meaning on average, Datafaker is about 4x faster than Javafaker.

Kotlin Faker does not support setting of birthday and blood. Without these 2 params it generates about 90M for 1 hour. net.datafaker.benchmark.generate_one_hour