In this episode of Brains & Beards, Patryk and Wojciech dive into why end-to-end tests often feel like “burning money” for mobile teams. They challenge the assumption that end-to-end tests provide reliable UI validation, arguing instead that they are expensive, flaky, and rarely catch meaningful issues. The hosts explain why UI-driven flows break easily, why maintenance costs balloon, and why the return on investment is often low except in specific cases such as performance tracking or automated screenshot generation for design reviews. They wrap up with pragmatic strategies for teams forced to use end-to-end tests and make the case that thoughtful architecture—and good communication—usually yield better results.
Key Moments
Defining end-to-end tests: The hosts clarify that true business end-to-end flows go far beyond what mobile teams can actually test, so the “ends” are usually artificially limited.
How they’re supposed to work: Tools like Detox or Maestro automate UI taps through accessibility labels, running flows on simulators or devices.
Why they fail: End-to-end tests are flaky, slow, expensive to run, and often break due to infrastructure issues rather than app logic.
High maintenance cost: Teams commonly spend 10–20% of engineering time on setup, debugging, and unreliable failures—often with little real value.
What they can be good for: Tracking performance (e.g., startup time) and generating automated screenshots for design review.
Why they don’t catch UI issues: Tests only verify tappability and flow—not layout, appearance, or subtle bugs.
If you must use them: Cache aggressively, learn your tool deeply, and write tests optimized for reliability, not full coverage.
Big picture: For most teams, end-to-end tests provide poor ROI; solid architecture and good communication usually solve the real problems better.
1
00:00:00,000 --> 00:00:04,250
(Music)
2
00:00:09,708 --> 00:00:11,791
Hello beautiful people, I'm Patrick.
3
00:00:12,500 --> 00:00:13,083
And I'm Hojek.
4
00:00:13,583 --> 00:00:15,250
And you are listening to
5
00:00:15,250 --> 00:00:16,250
Brains and Beards Show.
6
00:00:17,250 --> 00:00:19,666
Episode number 22, our
7
00:00:19,666 --> 00:00:22,000
end-to-end test, just burning money.
8
00:00:22,916 --> 00:00:23,291
Enjoy!
9
00:00:24,625 --> 00:00:27,875
(Music) Hello Hojek.
10
00:00:29,000 --> 00:00:29,583
Hello Patrick.
11
00:00:30,500 --> 00:00:30,958
How are you doing?
12
00:00:32,875 --> 00:00:35,291
Ah, it's a busy day.
13
00:00:36,500 --> 00:00:39,333
But it's always a great day when you get
14
00:00:39,333 --> 00:00:40,875
to complain about end-to-end tests.
15
00:00:41,166 --> 00:00:43,500
It's one of my favorite trends.
16
00:00:44,333 --> 00:00:46,916
Even better than the
17
00:00:46,916 --> 00:00:48,916
Netflix Witcher show?
18
00:00:49,666 --> 00:00:51,333
Oh, so much better, so much better.
19
00:00:52,041 --> 00:00:52,250
Oh.
20
00:00:52,708 --> 00:00:53,541
(Laughter)
21
00:00:55,208 --> 00:00:57,000
I didn't know that we're gonna
22
00:00:57,000 --> 00:00:59,625
talk about that kind of laws.
23
00:01:00,500 --> 00:01:02,208
But yeah, okay.
24
00:01:03,375 --> 00:01:07,791
I think it's in my personal top three
25
00:01:07,791 --> 00:01:10,458
personal pet peeves.
26
00:01:10,500 --> 00:01:12,708
Number one, unquestionably,
27
00:01:13,333 --> 00:01:15,208
the existence of dishwashers.
28
00:01:15,750 --> 00:01:17,250
I do not accept dishwashers.
29
00:01:18,000 --> 00:01:23,000
Second would be how the name of the
30
00:01:23,000 --> 00:01:24,250
United States of America
31
00:01:24,250 --> 00:01:26,708
is not properly normalized.
32
00:01:27,291 --> 00:01:28,500
Like it does not
33
00:01:28,500 --> 00:01:30,208
uniquely identify a country.
34
00:01:31,208 --> 00:01:33,500
And three is end-to-end tests.
35
00:01:34,208 --> 00:01:34,416
Okay.
36
00:01:34,541 --> 00:01:36,166
So obviously we're not gonna talk about
37
00:01:36,166 --> 00:01:39,958
dishwashers or the naming convention for
38
00:01:39,958 --> 00:01:41,666
countries in North America.
39
00:01:42,541 --> 00:01:43,875
So here we are.
40
00:01:44,375 --> 00:01:45,416
My third favorite topic.
41
00:01:46,416 --> 00:01:46,958
Okay.
42
00:01:47,791 --> 00:01:51,291
Okay, so maybe let's
43
00:01:51,291 --> 00:01:52,583
start with one thing.
44
00:01:52,791 --> 00:01:56,083
So we are on the same page.
45
00:01:56,083 --> 00:01:58,000
Like what is the end-to-end test?
46
00:01:58,375 --> 00:02:02,416
Because if you follow the definition, the
47
00:02:02,416 --> 00:02:04,000
end-to-end test is something which tests
48
00:02:04,000 --> 00:02:05,000
from the beginning to
49
00:02:05,000 --> 00:02:06,041
the end, the whole flow.
50
00:02:06,750 --> 00:02:09,333
So like if we are in the context of
51
00:02:09,333 --> 00:02:12,500
mobile applications, for me, if I would
52
00:02:12,500 --> 00:02:15,666
have an e-commerce site, that would be
53
00:02:15,666 --> 00:02:20,125
that I test on production because this is
54
00:02:20,125 --> 00:02:21,416
what's important to me.
55
00:02:21,666 --> 00:02:23,541
And I don't want to test staging for the
56
00:02:23,541 --> 00:02:27,166
developers to, I don't know, the people
57
00:02:27,166 --> 00:02:28,208
who are using the staging
58
00:02:28,208 --> 00:02:30,750
things to be sure that it works.
59
00:02:30,750 --> 00:02:32,208
I want to be sure that the
60
00:02:32,208 --> 00:02:33,625
production is working, I guess.
61
00:02:34,625 --> 00:02:38,083
And in that case, my flow would be the
62
00:02:38,083 --> 00:02:40,791
most important flow as an e-commerce
63
00:02:40,791 --> 00:02:42,875
owner would be that the person can
64
00:02:43,833 --> 00:02:46,875
register on mobile and with this account,
65
00:02:46,875 --> 00:02:49,166
they can search for the products I offer
66
00:02:49,166 --> 00:02:53,625
and they can purchase them using credit
67
00:02:53,625 --> 00:02:55,375
card because I think this is the most
68
00:02:55,375 --> 00:02:57,875
common way to buy things online.
69
00:02:58,875 --> 00:03:02,375
And I would be happy when the flow ends
70
00:03:02,375 --> 00:03:05,375
with like a confirmation of purchase.
71
00:03:07,208 --> 00:03:08,708
Are we going to talk about
72
00:03:08,708 --> 00:03:10,875
that or about something else?
73
00:03:11,666 --> 00:03:15,625
So yeah, first I'm going to make a
74
00:03:15,625 --> 00:03:19,541
philosophical remark because for you, the
75
00:03:19,541 --> 00:03:22,208
flow ends when the user makes a purchase.
76
00:03:22,416 --> 00:03:24,291
Yes, because I'm the owner of the site.
77
00:03:24,500 --> 00:03:26,541
That's what's important to me.
78
00:03:27,375 --> 00:03:33,416
Well, actually, if we were a video
79
00:03:33,416 --> 00:03:36,916
medium, we could put the meme in.
80
00:03:37,750 --> 00:03:39,291
But the problem is that the
81
00:03:39,291 --> 00:03:41,208
user does not really care.
82
00:03:42,958 --> 00:03:45,708
Like for them, the flow does not end when
83
00:03:45,708 --> 00:03:46,583
they make a purchase.
84
00:03:47,208 --> 00:03:49,541
The flow ends after the product, the
85
00:03:49,541 --> 00:03:51,750
purchase is registered in the warehouse.
86
00:03:52,625 --> 00:03:56,458
The product is packaged, shipped to him.
87
00:03:56,958 --> 00:04:00,000
He actually gets it from the messenger
88
00:04:00,000 --> 00:04:02,666
and there is a correct invoice
89
00:04:02,666 --> 00:04:05,000
inside and the product works.
90
00:04:05,291 --> 00:04:08,125
Or if it doesn't work, they can return it
91
00:04:08,125 --> 00:04:09,041
in a comfortable way.
92
00:04:09,625 --> 00:04:12,791
This is real end to end from
93
00:04:12,791 --> 00:04:14,041
the business perspective, right?
94
00:04:15,041 --> 00:04:17,750
So we hit the first.
95
00:04:17,750 --> 00:04:20,250
That's not even end because then you have
96
00:04:20,250 --> 00:04:25,000
like the 14 days of policy of returning.
97
00:04:26,041 --> 00:04:28,833
And then you have to pay the taxes.
98
00:04:29,250 --> 00:04:31,125
I mean, like you are right.
99
00:04:31,125 --> 00:04:33,541
We have to put some boundaries.
100
00:04:33,541 --> 00:04:35,041
We have to put the end somewhere.
101
00:04:36,458 --> 00:04:39,333
And I think how you described it in the
102
00:04:39,333 --> 00:04:41,416
beginning, it's a very common approach
103
00:04:41,416 --> 00:04:45,625
where we put the ends when there is the
104
00:04:45,625 --> 00:04:48,666
end of responsibility of a particular
105
00:04:48,666 --> 00:04:51,041
team that builds the end to end test.
106
00:04:51,750 --> 00:04:53,833
So for example, if you have a mobile
107
00:04:53,833 --> 00:04:55,958
development team, the team that only
108
00:04:55,958 --> 00:04:57,125
deals with the mobile app,
109
00:04:58,166 --> 00:05:00,208
you will probably mock out the network
110
00:05:00,208 --> 00:05:02,708
traffic because you cannot control it.
111
00:05:02,916 --> 00:05:03,916
Like that's not that's
112
00:05:03,916 --> 00:05:05,666
outside of the scope of your team.
113
00:05:06,250 --> 00:05:08,166
And even if you wanted to
114
00:05:08,166 --> 00:05:10,291
have it not mocked, you can't.
115
00:05:10,958 --> 00:05:12,000
Because you're not the one
116
00:05:12,000 --> 00:05:13,291
responsible for the servers.
117
00:05:13,750 --> 00:05:16,125
So you cannot make fix your test when
118
00:05:16,125 --> 00:05:18,291
they start failing because network.
119
00:05:19,041 --> 00:05:20,041
So you would mock them out.
120
00:05:20,500 --> 00:05:23,000
But on the other hand, if you work as a
121
00:05:23,000 --> 00:05:25,375
team of full stack developers that also
122
00:05:25,375 --> 00:05:26,708
build the mobile app,
123
00:05:27,791 --> 00:05:29,208
you could have the real traffic, you
124
00:05:29,208 --> 00:05:29,833
could spin up the
125
00:05:29,833 --> 00:05:30,958
servers if you want to write.
126
00:05:32,000 --> 00:05:36,458
So I think we make a decision here, which
127
00:05:36,458 --> 00:05:38,708
is already controversial decision for
128
00:05:38,708 --> 00:05:43,291
many people, that we put the ends where
129
00:05:43,291 --> 00:05:46,000
the responsibilities of the mobile team
130
00:05:46,000 --> 00:05:48,583
start and where they end, right?
131
00:05:49,375 --> 00:05:52,458
Yes, otherwise you put in kind of a big
132
00:05:52,458 --> 00:05:55,083
burden on your mobile team that they're
133
00:05:55,083 --> 00:05:58,125
starting to be like kind of police force
134
00:05:58,125 --> 00:06:01,416
in the company where they have to run
135
00:06:01,416 --> 00:06:04,000
from team to team and convince them to
136
00:06:04,000 --> 00:06:06,375
fix something because they cannot do it
137
00:06:06,375 --> 00:06:07,500
on their own as they
138
00:06:07,500 --> 00:06:08,458
don't have expertise.
139
00:06:08,500 --> 00:06:12,000
Access to the things which got broken
140
00:06:12,000 --> 00:06:13,666
outside of the mobile application.
141
00:06:14,416 --> 00:06:17,833
And that's the worst thing to burn the
142
00:06:17,833 --> 00:06:21,000
time from the people who should focus,
143
00:06:22,416 --> 00:06:24,000
from the people who should focus on
144
00:06:24,000 --> 00:06:26,541
working on mobile app and not policing
145
00:06:26,541 --> 00:06:29,500
the company about failing tests.
146
00:06:30,291 --> 00:06:32,708
And also that's a very toxic situation to
147
00:06:32,708 --> 00:06:37,250
be in to where your work is to the first
148
00:06:37,250 --> 00:06:39,708
90% of what you do is verify that
149
00:06:39,708 --> 00:06:42,000
everybody else did their work correctly.
150
00:06:42,250 --> 00:06:44,583
And if they didn't go and complain, like
151
00:06:44,583 --> 00:06:47,541
that's if I wanted to do
152
00:06:47,541 --> 00:06:48,958
it, I'll be doing QA, right?
153
00:06:49,708 --> 00:06:49,958
Exactly.
154
00:06:51,208 --> 00:06:53,250
So yeah, so this is the first limitation
155
00:06:53,250 --> 00:06:54,375
that end to ends are not
156
00:06:54,375 --> 00:06:55,625
really end to end, right?
157
00:06:55,625 --> 00:06:55,916
Because
158
00:06:57,375 --> 00:07:00,625
probably the CEO that says he wants to
159
00:07:00,625 --> 00:07:04,000
have end to end tests or CTO does not
160
00:07:04,000 --> 00:07:07,458
mean the ends like the team would
161
00:07:07,458 --> 00:07:09,000
interpret it, right?
162
00:07:12,541 --> 00:07:14,875
So, but yeah, let's limit it.
163
00:07:15,208 --> 00:07:16,208
We mock the network.
164
00:07:16,583 --> 00:07:19,458
We only control our own stuff.
165
00:07:22,416 --> 00:07:26,958
Maybe you can start us off describing
166
00:07:26,958 --> 00:07:28,166
like, how would you go
167
00:07:28,166 --> 00:07:29,333
about implementing it?
168
00:07:29,666 --> 00:07:32,541
Like, what's the optimistic scenario how
169
00:07:32,541 --> 00:07:34,708
this implementation could go?
170
00:07:36,250 --> 00:07:37,000
Optimistic.
171
00:07:38,166 --> 00:07:40,416
To be honest, I don't have many
172
00:07:40,416 --> 00:07:42,708
optimistic cases from my
173
00:07:42,708 --> 00:07:45,041
experience in the way that
174
00:07:46,416 --> 00:07:48,583
the tests when they are set up, they kind
175
00:07:48,583 --> 00:07:50,583
of work, but not for the long time.
176
00:07:51,708 --> 00:07:53,000
And doesn't matter if you
177
00:07:53,000 --> 00:07:54,958
mock something out or not.
178
00:07:55,500 --> 00:07:57,833
But you know, if you read the
179
00:07:57,833 --> 00:08:00,375
documentation, I guess that would
180
00:08:00,375 --> 00:08:04,625
describe the positive cases that you use,
181
00:08:04,625 --> 00:08:06,458
I guess detox on my stroke, perhaps
182
00:08:06,500 --> 00:08:09,708
today, which kind of instruments your
183
00:08:09,708 --> 00:08:14,833
computer to start at the simulator or
184
00:08:14,833 --> 00:08:16,833
emulator depends if you run
185
00:08:16,833 --> 00:08:20,500
them on iPhone or Android.
186
00:08:20,958 --> 00:08:24,166
And then you use this accessibility
187
00:08:24,166 --> 00:08:27,500
labels to kind of start the application
188
00:08:27,500 --> 00:08:28,708
and then click around.
189
00:08:30,041 --> 00:08:36,250
And if the flow cannot be succeed, then
190
00:08:36,250 --> 00:08:37,333
you have failed tests.
191
00:08:37,625 --> 00:08:40,666
And if, you know, like the test can
192
00:08:40,666 --> 00:08:42,375
navigate from the beginning to the end
193
00:08:42,375 --> 00:08:47,125
and accomplish what it is supposed to
194
00:08:47,125 --> 00:08:49,125
accomplish, then you have a green test.
195
00:08:49,708 --> 00:08:50,625
But that's, you know, like,
196
00:08:50,916 --> 00:08:52,250
this is only in documentation.
197
00:08:52,916 --> 00:08:54,000
And perhaps it works on
198
00:08:54,000 --> 00:08:56,458
the first test you write.
199
00:08:56,833 --> 00:08:58,500
And perhaps it works for the first week
200
00:08:58,500 --> 00:09:00,041
with the whole test suite.
201
00:09:00,791 --> 00:09:03,541
But on some point, there are problems
202
00:09:03,583 --> 00:09:08,125
because you don't control basically the
203
00:09:08,125 --> 00:09:11,500
domain where you were there are different
204
00:09:11,500 --> 00:09:13,250
ways why it doesn't work.
205
00:09:13,583 --> 00:09:15,583
But but that would be, you know, like,
206
00:09:15,583 --> 00:09:18,333
perhaps we can focus on what doesn't work
207
00:09:18,333 --> 00:09:24,000
and why later on, but the implementation
208
00:09:24,000 --> 00:09:25,333
will be like that, you
209
00:09:25,333 --> 00:09:27,166
know, using detox on my show.
210
00:09:28,125 --> 00:09:30,916
And you would run it, you can run it in
211
00:09:30,916 --> 00:09:32,375
different places, right?
212
00:09:32,375 --> 00:09:33,041
There are services
213
00:09:33,041 --> 00:09:34,625
which allows you to run
214
00:09:36,166 --> 00:09:37,125
those tests.
215
00:09:38,791 --> 00:09:41,250
It that gets expensive pretty fast.
216
00:09:42,166 --> 00:09:45,916
But you can as well use it like my
217
00:09:45,916 --> 00:09:47,875
computer in your office and run it there,
218
00:09:48,583 --> 00:09:52,500
which is cheaper, but comes with its own
219
00:09:52,500 --> 00:09:54,291
costs, basically, and we
220
00:09:54,291 --> 00:09:55,458
can talk about them later.
221
00:09:56,583 --> 00:09:59,708
So we have the happy scenario, like, we
222
00:09:59,708 --> 00:10:01,958
know how, in principle, it's supposed to
223
00:10:01,958 --> 00:10:04,291
work like you write some testing
224
00:10:04,291 --> 00:10:08,458
scenarios in detox or maestro, or in
225
00:10:08,458 --> 00:10:10,083
Appium, if you really hate your life.
226
00:10:11,166 --> 00:10:13,000
And, and you run it
227
00:10:13,000 --> 00:10:15,250
against Android and iOS apps.
228
00:10:16,041 --> 00:10:20,083
And you use the accessibility label to
229
00:10:20,083 --> 00:10:22,583
click through stuff and hopefully get to
230
00:10:22,583 --> 00:10:23,750
the end of the scenario.
231
00:10:24,500 --> 00:10:27,583
So as you mentioned, there's like a lot of
232
00:10:27,583 --> 00:10:29,500
things that can go go wrong there.
233
00:10:30,291 --> 00:10:35,750
And let's let's talk about the traps that
234
00:10:35,750 --> 00:10:36,625
are waiting for you.
235
00:10:37,416 --> 00:10:41,083
I think the first one is that
236
00:10:42,625 --> 00:10:46,916
they are very expensive in two ways.
237
00:10:47,125 --> 00:10:49,250
One is you mentioned the machines that
238
00:10:49,250 --> 00:10:50,583
they run on, because
239
00:10:50,583 --> 00:10:51,875
you have to run the build.
240
00:10:52,916 --> 00:10:55,000
And you have to the
241
00:10:55,000 --> 00:10:57,083
tests take a while to run.
242
00:10:57,083 --> 00:10:59,125
And if you'd like to do it on every
243
00:10:59,125 --> 00:11:01,666
comment or every push or PR or whatever,
244
00:11:03,625 --> 00:11:06,000
or parallelize, yeah, it adds up.
245
00:11:06,750 --> 00:11:09,583
And also, if you want to cover iOS, which
246
00:11:09,583 --> 00:11:13,125
probably you would like to, then Mac
247
00:11:13,125 --> 00:11:15,500
machines on like remote servers are much
248
00:11:15,500 --> 00:11:18,208
more expensive than the
249
00:11:18,208 --> 00:11:19,250
Linux machines, right?
250
00:11:20,583 --> 00:11:23,416
And so one way around it is, as you
251
00:11:23,416 --> 00:11:26,833
mentioned, to just buy a 700 bucks.
252
00:11:28,291 --> 00:11:32,166
You pay 700 bucks for a Mac Mini, and you
253
00:11:32,166 --> 00:11:33,208
put it in your office.
254
00:11:34,208 --> 00:11:36,416
But that's another computer on your local
255
00:11:36,416 --> 00:11:39,958
network that your team has to secure.
256
00:11:40,625 --> 00:11:42,500
Because it has to be accessible from
257
00:11:42,500 --> 00:11:45,208
outside like there, there is the extra
258
00:11:45,208 --> 00:11:47,083
management cost of this.
259
00:11:47,083 --> 00:11:48,750
And that's the expertise your team might
260
00:11:48,750 --> 00:11:50,000
have or might not have, right?
261
00:11:51,125 --> 00:11:53,541
But there's also something that I call
262
00:11:53,541 --> 00:11:58,916
the end to end tax, which means that
263
00:11:58,916 --> 00:12:04,333
testing UI is inherently complicated and
264
00:12:04,333 --> 00:12:06,916
flaky, like it's, it's hard to do you're
265
00:12:06,916 --> 00:12:08,416
dealing with so many layers, so many
266
00:12:08,416 --> 00:12:12,875
things can go wrong, that it's so much
267
00:12:12,875 --> 00:12:14,833
easier to just write a unit test for a
268
00:12:14,833 --> 00:12:17,291
function that takes some kind of input
269
00:12:17,291 --> 00:12:19,000
and produces some kind of output.
270
00:12:20,083 --> 00:12:21,500
And with UI, it's
271
00:12:21,500 --> 00:12:23,041
never as simple as that.
272
00:12:24,000 --> 00:12:26,583
And end to end tests are particularly
273
00:12:26,583 --> 00:12:30,875
flaky, because you don't isolate the
274
00:12:30,875 --> 00:12:32,500
thing you're testing, but you're kind of
275
00:12:32,500 --> 00:12:34,458
running everything at the same time.
276
00:12:35,500 --> 00:12:39,083
And this end to end tax that I call it,
277
00:12:40,000 --> 00:12:44,000
it's often like as high as like 20% of
278
00:12:44,000 --> 00:12:48,958
the effort of the team is spent on end to
279
00:12:48,958 --> 00:12:50,291
end tests for the
280
00:12:50,291 --> 00:12:51,916
features that they're building.
281
00:12:52,916 --> 00:12:54,750
In the first months, they're working with
282
00:12:54,750 --> 00:12:56,958
the with the end to end test suite.
283
00:12:58,041 --> 00:13:00,666
And later it goes it goes down, but it's
284
00:13:00,666 --> 00:13:03,333
still I think, like around 10%.
285
00:13:03,458 --> 00:13:07,583
Even when you have an established end to
286
00:13:07,583 --> 00:13:10,041
end test suite that is running correctly,
287
00:13:10,041 --> 00:13:12,208
and you set up the infrastructure, and
288
00:13:12,208 --> 00:13:13,416
everything seems to be fine.
289
00:13:14,500 --> 00:13:19,041
And 20% on a five person team is one
290
00:13:19,041 --> 00:13:22,333
person full time dedicated to your having
291
00:13:22,333 --> 00:13:23,375
those end to end tests.
292
00:13:23,916 --> 00:13:26,250
And that's, it's a
293
00:13:26,250 --> 00:13:27,583
huge price to pay, right?
294
00:13:27,583 --> 00:13:31,291
It doesn't because later it will go down
295
00:13:31,291 --> 00:13:32,958
to this person's half time
296
00:13:32,958 --> 00:13:34,416
spent on end to end test.
297
00:13:35,166 --> 00:13:39,250
But there's no other feature that you
298
00:13:39,250 --> 00:13:42,166
would be happy like paying this high
299
00:13:42,166 --> 00:13:43,791
ongoing cost for like
300
00:13:43,791 --> 00:13:45,083
just maintaining right.
301
00:13:45,833 --> 00:13:48,666
Yeah, that's, that's actually what we
302
00:13:48,666 --> 00:13:52,458
have in one of the teams is like the one
303
00:13:52,458 --> 00:13:55,083
person was almost the full time working
304
00:13:55,083 --> 00:13:58,708
on infrastructure of of
305
00:13:58,708 --> 00:14:01,625
having off of the end to end test.
306
00:14:01,625 --> 00:14:04,666
Because in that case, there was the Mac
307
00:14:04,666 --> 00:14:07,875
mini involved, which had to be connected
308
00:14:07,875 --> 00:14:11,958
to the network and updated periodically.
309
00:14:13,000 --> 00:14:16,083
And things were working for a week and
310
00:14:16,083 --> 00:14:18,333
then it did it think
311
00:14:18,333 --> 00:14:19,875
stops working and so on.
312
00:14:20,708 --> 00:14:22,791
But to be honest, it's like as well.
313
00:14:24,458 --> 00:14:28,666
Like I in my life, I never had and to end
314
00:14:28,666 --> 00:14:32,000
test finding like really valuable stuff.
315
00:14:33,791 --> 00:14:37,625
So even in that case, where they did the
316
00:14:37,625 --> 00:14:40,000
company put a lot of money on it, like
317
00:14:40,000 --> 00:14:42,666
having one person almost full time
318
00:14:42,666 --> 00:14:45,375
working on it, it never found anything.
319
00:14:46,583 --> 00:14:49,416
And like, if you think about it, like
320
00:14:49,416 --> 00:14:52,583
when would you when would you think that
321
00:14:52,583 --> 00:14:55,958
end to end test would give you some value
322
00:14:55,958 --> 00:14:58,833
is that is the case when you have like
323
00:14:58,833 --> 00:15:03,458
big team, actually, even multiple teams,
324
00:15:03,666 --> 00:15:06,416
I would say, and you think that the team
325
00:15:06,416 --> 00:15:07,541
doesn't communicate
326
00:15:07,541 --> 00:15:09,958
between each other very well.
327
00:15:09,958 --> 00:15:11,000
And perhaps you have a lot of
328
00:15:11,041 --> 00:15:12,833
microservices and so on.
329
00:15:13,416 --> 00:15:15,208
And you would would like to test this
330
00:15:15,208 --> 00:15:19,500
because perhaps one part of the team can
331
00:15:19,500 --> 00:15:21,000
break something for somebody else.
332
00:15:21,500 --> 00:15:24,250
And perhaps they didn't manage to test it
333
00:15:24,250 --> 00:15:26,666
because perhaps that what they broke was
334
00:15:26,666 --> 00:15:29,458
behind some obscure use case.
335
00:15:29,916 --> 00:15:33,333
And because it's not that they're part of
336
00:15:34,333 --> 00:15:36,083
their team to work on it, they don't know
337
00:15:36,083 --> 00:15:37,583
about this use case and so on.
338
00:15:37,958 --> 00:15:39,458
So you would like to have like, I guess,
339
00:15:39,750 --> 00:15:42,000
a big suite of tests, which they try to
340
00:15:42,000 --> 00:15:43,708
test all that stuff to find it.
341
00:15:44,250 --> 00:15:48,041
But then probably, you would have to
342
00:15:48,041 --> 00:15:50,166
really test it against an API.
343
00:15:50,916 --> 00:15:52,958
Because I don't think like just testing
344
00:15:52,958 --> 00:15:56,125
your eyes is would help you in that case.
345
00:15:57,083 --> 00:15:59,458
And we already established that testing
346
00:15:59,458 --> 00:16:02,041
like from beginning to the end is super
347
00:16:02,041 --> 00:16:05,208
complex and and cost a lot
348
00:16:05,208 --> 00:16:06,625
of money and you are not.
349
00:16:07,625 --> 00:16:09,458
And then you have the problem who owns
350
00:16:09,458 --> 00:16:10,833
the test and who has the
351
00:16:10,833 --> 00:16:13,291
right to to fix them and so on.
352
00:16:13,583 --> 00:16:15,000
And it's like a mess.
353
00:16:15,250 --> 00:16:16,083
Like, I don't think
354
00:16:16,083 --> 00:16:18,000
testing UI is very valuable.
355
00:16:18,916 --> 00:16:22,375
Well, in the scenario you describe, we go
356
00:16:22,375 --> 00:16:24,625
back to the basic principle where where
357
00:16:24,625 --> 00:16:28,208
are the ends and the ends are when where
358
00:16:28,208 --> 00:16:30,250
the team, where the
359
00:16:30,250 --> 00:16:31,875
team's responsibility finishes.
360
00:16:32,750 --> 00:16:34,500
So if you want to end to end test
361
00:16:34,500 --> 00:16:37,666
something across several teams, then you
362
00:16:37,666 --> 00:16:43,000
have to spin up spin up a new team whose
363
00:16:43,000 --> 00:16:45,708
responsibility is solely doing the end to
364
00:16:45,708 --> 00:16:47,250
end test between those teams.
365
00:16:47,250 --> 00:16:49,291
And you have like an infrastructure team.
366
00:16:49,750 --> 00:16:51,750
If you're in a situation where you see
367
00:16:51,750 --> 00:16:54,083
return on investment from this.
368
00:16:54,750 --> 00:16:58,833
And sure, I guess that's one expensive
369
00:16:58,833 --> 00:17:00,458
way of solving this problem other than
370
00:17:00,458 --> 00:17:02,083
having people talk to each other.
371
00:17:02,708 --> 00:17:05,041
But yeah, but that's the
372
00:17:05,875 --> 00:17:07,375
where the title comes from.
373
00:17:08,375 --> 00:17:09,666
That's the burning money scenario.
374
00:17:10,083 --> 00:17:11,666
It's a it's a way to solve the problem.
375
00:17:11,666 --> 00:17:12,666
It's an expensive way.
376
00:17:13,125 --> 00:17:14,791
It kind of work, but
377
00:17:14,791 --> 00:17:16,416
it's a pain in the butt.
378
00:17:17,291 --> 00:17:19,416
Because just to mention, because on the
379
00:17:19,416 --> 00:17:21,000
other hand, we have, you know, like a
380
00:17:21,000 --> 00:17:25,541
small team and a small company in that
381
00:17:25,541 --> 00:17:28,291
case, I don't think you would have that
382
00:17:28,291 --> 00:17:31,750
complicated application that you couldn't
383
00:17:31,750 --> 00:17:36,625
test it manually, or and additionally
384
00:17:36,625 --> 00:17:40,000
have a lot of like, very fast tests, as
385
00:17:40,000 --> 00:17:42,375
you said, like the unit test or the
386
00:17:42,375 --> 00:17:44,250
functional test, this kind of things.
387
00:17:44,833 --> 00:17:48,333
So I agree, I think end to end tests.
388
00:17:49,375 --> 00:17:52,250
While they are a useful tool, they are
389
00:17:52,250 --> 00:17:53,791
just not useful for testing.
390
00:17:54,291 --> 00:17:57,875
Like, I think they have their use cases,
391
00:17:58,000 --> 00:18:00,625
which we'll talk about later.
392
00:18:01,583 --> 00:18:03,666
But it's, they are not
393
00:18:03,666 --> 00:18:05,625
really for for testing.
394
00:18:07,708 --> 00:18:07,916
And
395
00:18:09,833 --> 00:18:12,083
but let's let's back up a bit.
396
00:18:12,375 --> 00:18:14,666
Because we mentioned that this flaky and
397
00:18:14,666 --> 00:18:16,541
cumbersome and difficult to maintain.
398
00:18:16,541 --> 00:18:20,166
But why is it flaky?
399
00:18:20,166 --> 00:18:21,083
Why is cumbersome?
400
00:18:21,083 --> 00:18:22,375
Why is it difficult to to
401
00:18:22,375 --> 00:18:25,125
keep them up to date and passing?
402
00:18:26,625 --> 00:18:31,833
One of the things is that, again, this is
403
00:18:31,833 --> 00:18:33,916
the case, if you want to burn the money
404
00:18:33,916 --> 00:18:37,000
and pay somebody for managing the
405
00:18:37,000 --> 00:18:38,833
infrastructure for you in the cloud,
406
00:18:39,208 --> 00:18:42,416
like, I don't know, export, GitHub
407
00:18:42,416 --> 00:18:44,125
actions, and so on, this kind of things,
408
00:18:45,208 --> 00:18:48,250
then it's much easier, because they are
409
00:18:48,250 --> 00:18:50,250
kind of responsible for making sure that
410
00:18:50,250 --> 00:18:53,083
the simulators are starting and so on.
411
00:18:53,541 --> 00:18:57,208
But on some point, even in that case, you
412
00:18:57,208 --> 00:19:00,041
will have errors like, well, I couldn't
413
00:19:00,041 --> 00:19:02,541
find an element, or well, the simulator
414
00:19:02,625 --> 00:19:05,833
didn't start or did there gonna be a lot
415
00:19:05,833 --> 00:19:08,458
of cases where things are not working,
416
00:19:08,458 --> 00:19:10,208
and they don't have anything to do with
417
00:19:10,208 --> 00:19:12,541
your mobile application even.
418
00:19:13,500 --> 00:19:15,375
And this is like somebody has to go go
419
00:19:15,375 --> 00:19:17,000
through the logs, find
420
00:19:17,000 --> 00:19:18,500
out what doesn't work.
421
00:19:19,166 --> 00:19:21,208
If that's if something breaks around
422
00:19:21,208 --> 00:19:23,375
switching of iOS versions,
423
00:19:24,541 --> 00:19:26,166
then probably you don't
424
00:19:26,166 --> 00:19:28,125
perhaps know what is missing.
425
00:19:28,125 --> 00:19:30,125
And then you start googling and perhaps
426
00:19:30,958 --> 00:19:34,541
you will have to wait for a week or two
427
00:19:34,541 --> 00:19:37,333
till somebody finds a solution for this.
428
00:19:38,000 --> 00:19:39,500
And you know, this is the period of the
429
00:19:39,500 --> 00:19:41,125
time where you have end to end tests, but
430
00:19:41,125 --> 00:19:42,208
they are not working because
431
00:19:42,958 --> 00:19:44,500
something else doesn't work.
432
00:19:44,833 --> 00:19:46,416
And it's not your mobile application.
433
00:19:46,916 --> 00:19:47,583
And what do you do
434
00:19:47,583 --> 00:19:48,708
with those red PRs, right?
435
00:19:49,541 --> 00:19:50,000
Yeah, exactly.
436
00:19:50,291 --> 00:19:54,291
So you're starting to have to have a lot
437
00:19:54,291 --> 00:19:56,541
of broken windows, basically, the
438
00:19:56,541 --> 00:19:58,375
principle, you know, commented out
439
00:19:58,375 --> 00:20:00,666
there's tests failing, and everybody
440
00:20:00,666 --> 00:20:03,833
says, you know, everybody is gonna say,
441
00:20:03,833 --> 00:20:05,875
Okay, for now, we don't care about this,
442
00:20:05,875 --> 00:20:07,166
you know, we're gonna care about it in
443
00:20:07,166 --> 00:20:08,375
one week or in two weeks.
444
00:20:08,666 --> 00:20:11,583
And on some point, you know, we all been
445
00:20:11,583 --> 00:20:15,208
there probably, and that you open a
446
00:20:15,208 --> 00:20:18,458
project and you run TSC, and then you
447
00:20:18,458 --> 00:20:21,000
get, I don't know 1000 warnings.
448
00:20:21,875 --> 00:20:24,375
And yes, Lin throws, I
449
00:20:24,375 --> 00:20:25,958
don't know, 100 errors, right?
450
00:20:25,958 --> 00:20:28,000
Because on some points, one person
451
00:20:28,000 --> 00:20:31,375
stopped carrying and then everybody else
452
00:20:31,375 --> 00:20:33,166
copied this attitude.
453
00:20:33,500 --> 00:20:37,750
And then you end up in the bad place.
454
00:20:38,833 --> 00:20:39,041
Yeah.
455
00:20:39,583 --> 00:20:43,458
And also, it's important to mention the
456
00:20:43,458 --> 00:20:46,208
scenarios that you pointed out, like
457
00:20:46,208 --> 00:20:48,958
there, for some reason, an element could
458
00:20:48,958 --> 00:20:52,375
not be found or like an emulator flows,
459
00:20:53,083 --> 00:20:56,541
those hiccups, even if they happen once
460
00:20:56,541 --> 00:21:01,083
every 100 test runs, like 1% of the time,
461
00:21:01,916 --> 00:21:05,000
if you have 10 tests running in your test
462
00:21:05,000 --> 00:21:08,541
suite, you're suddenly at like 10%
463
00:21:09,000 --> 00:21:11,541
failing rate, like every 10th run of the
464
00:21:11,541 --> 00:21:16,291
test suite has a chance of one of the of
465
00:21:16,291 --> 00:21:17,833
one of the tests randomly failing.
466
00:21:18,375 --> 00:21:20,250
And that's a lot like it means like every
467
00:21:20,250 --> 00:21:23,208
10th PR, you have to like jump in and
468
00:21:23,208 --> 00:21:24,916
figure out what's, what's
469
00:21:24,916 --> 00:21:27,875
happening or rerun this test and
470
00:21:29,458 --> 00:21:31,541
they would take typically somewhere
471
00:21:31,541 --> 00:21:33,458
between 10 minutes and an hour.
472
00:21:34,208 --> 00:21:36,708
So what are you going to do this time 10
473
00:21:36,708 --> 00:21:40,708
minutes is dangerous, Twitter category
474
00:21:40,708 --> 00:21:43,500
category, with one hour, you would
475
00:21:43,500 --> 00:21:45,250
probably switch to a different task and
476
00:21:45,250 --> 00:21:46,541
suddenly you end up working
477
00:21:46,541 --> 00:21:48,000
on two things at the same time.
478
00:21:48,791 --> 00:21:51,833
So, uh, yeah, it's a, there is a lot of
479
00:21:51,833 --> 00:21:56,625
team general teams, lowdowns coming from
480
00:21:56,625 --> 00:21:59,375
seemingly very rare mistakes, right?
481
00:22:00,375 --> 00:22:03,625
Like one error in 100 runs, it does not
482
00:22:03,625 --> 00:22:05,958
seem like a lot, but suddenly it spirals
483
00:22:05,958 --> 00:22:07,666
out of out of control.
484
00:22:07,916 --> 00:22:08,500
Yeah.
485
00:22:08,500 --> 00:22:09,500
And sometimes you, you
486
00:22:09,500 --> 00:22:11,083
know, you can get a snapshot.
487
00:22:11,375 --> 00:22:12,583
They're showing you
488
00:22:12,583 --> 00:22:15,083
where the test failed.
489
00:22:15,416 --> 00:22:17,458
And then you clearly, you can see the
490
00:22:17,458 --> 00:22:20,625
button which the test couldn't find it.
491
00:22:20,625 --> 00:22:23,083
And then, you know, start people are
492
00:22:23,083 --> 00:22:24,833
starting to coming up with ideas.
493
00:22:24,833 --> 00:22:25,166
Okay.
494
00:22:25,166 --> 00:22:26,583
We had some sleep time
495
00:22:26,583 --> 00:22:27,583
or something like that.
496
00:22:27,583 --> 00:22:30,750
And, and, and then you test starting to
497
00:22:30,750 --> 00:22:32,208
get more and more bizarre.
498
00:22:33,125 --> 00:22:33,625
All right.
499
00:22:33,625 --> 00:22:35,291
Scroll the list a little bit farther.
500
00:22:35,666 --> 00:22:36,416
Yeah, exactly.
501
00:22:36,416 --> 00:22:39,666
A few pixels to the left, and a little
502
00:22:39,666 --> 00:22:40,791
bit more to the right.
503
00:22:41,000 --> 00:22:42,291
And maybe then it works.
504
00:22:44,708 --> 00:22:48,791
I mean, like this person who wrote it
505
00:22:48,791 --> 00:22:50,541
probably knows what is going on.
506
00:22:50,541 --> 00:22:52,500
The next person who never saw this test,
507
00:22:52,500 --> 00:22:54,916
they gonna scratch your head, scratch
508
00:22:54,916 --> 00:22:56,583
their head and, you know,
509
00:22:58,000 --> 00:22:58,750
asking themselves.
510
00:22:59,125 --> 00:23:01,291
So if they shouldn't switch the job.
511
00:23:02,208 --> 00:23:03,250
Okay.
512
00:23:03,625 --> 00:23:06,041
Let's go back to the issue of money.
513
00:23:06,958 --> 00:23:12,333
So if we feel that it's burning money,
514
00:23:12,750 --> 00:23:13,583
and we have mentioned
515
00:23:13,583 --> 00:23:15,125
this phrase a few times,
516
00:23:16,500 --> 00:23:18,541
it's important to put it in perspective
517
00:23:18,541 --> 00:23:23,458
on what is the, what's the
518
00:23:23,458 --> 00:23:24,708
return on the investment, right?
519
00:23:24,708 --> 00:23:26,833
Because nobody's like burning money to
520
00:23:26,833 --> 00:23:29,125
just throw it away, but they are spending
521
00:23:29,125 --> 00:23:31,666
money in hopes of getting some results.
522
00:23:32,416 --> 00:23:34,333
And let's talk about what results are
523
00:23:34,333 --> 00:23:37,458
reasonable and what are unreasonable.
524
00:23:38,250 --> 00:23:39,916
And I think we already mentioned that
525
00:23:39,916 --> 00:23:44,333
testing UI with end-to-end tests is,
526
00:23:45,375 --> 00:23:46,916
it's an unreasonable expectation.
527
00:23:47,166 --> 00:23:51,541
You cannot say that, Hey, I have an suite
528
00:23:51,541 --> 00:23:52,750
of end-to-end tests.
529
00:23:53,500 --> 00:23:55,250
And this means that if there are any
530
00:23:55,250 --> 00:23:56,458
visual glitches in my
531
00:23:56,458 --> 00:23:58,041
app, I will, I will see them.
532
00:23:58,875 --> 00:24:00,000
No, you won't.
533
00:24:00,291 --> 00:24:03,875
Like, like you're, you're, you wrote a
534
00:24:03,875 --> 00:24:06,291
script to tap on accessibility labels.
535
00:24:06,791 --> 00:24:07,916
You don't even know what
536
00:24:07,916 --> 00:24:09,208
color your buttons are.
537
00:24:09,583 --> 00:24:11,083
You don't, sometimes you don't even,
538
00:24:11,208 --> 00:24:12,125
you're not even sure
539
00:24:12,125 --> 00:24:13,000
that they are visible.
540
00:24:13,291 --> 00:24:14,083
They are not like,
541
00:24:14,083 --> 00:24:16,166
hidden partially by something.
542
00:24:16,791 --> 00:24:18,625
As long as there is a pixel that can be
543
00:24:18,625 --> 00:24:20,333
found and tapped, sure,
544
00:24:20,333 --> 00:24:21,583
your test is passing, right?
545
00:24:22,291 --> 00:24:25,166
So we cannot expect to find UI issues.
546
00:24:25,416 --> 00:24:26,750
So what, what can we find?
547
00:24:27,291 --> 00:24:31,416
There are two scenarios where I actually
548
00:24:31,416 --> 00:24:33,208
like end-to-end test.
549
00:24:34,000 --> 00:24:36,416
One is tracking performance.
550
00:24:37,458 --> 00:24:40,875
Because if you want to have a tool to see
551
00:24:40,875 --> 00:24:42,000
how the performance of
552
00:24:42,000 --> 00:24:43,958
your app changes over time,
553
00:24:45,375 --> 00:24:47,208
you have to benchmark it on the same
554
00:24:47,208 --> 00:24:50,500
machine under the same conditions and
555
00:24:50,500 --> 00:24:51,708
keep the locks of it.
556
00:24:52,000 --> 00:24:55,708
So you cannot rely on your developers,
557
00:24:56,041 --> 00:24:58,625
everybody benchmarking their features or
558
00:24:58,625 --> 00:25:00,541
their PRs on their machines.
559
00:25:01,041 --> 00:25:02,208
You'd have to do it somewhere.
560
00:25:02,208 --> 00:25:04,291
And benchmarking means you have to have a
561
00:25:04,291 --> 00:25:05,500
real build of your app.
562
00:25:05,500 --> 00:25:06,458
You have to run it and you
563
00:25:06,458 --> 00:25:07,541
have to do something on it.
564
00:25:08,458 --> 00:25:12,708
And I think one of the important
565
00:25:12,708 --> 00:25:14,875
performance benchmarks is the startup
566
00:25:14,875 --> 00:25:16,333
time, which you could
567
00:25:16,333 --> 00:25:18,041
measure with the end-to-end test.
568
00:25:18,708 --> 00:25:19,750
And actually like there
569
00:25:19,750 --> 00:25:21,000
are cool tools for that.
570
00:25:21,041 --> 00:25:23,500
You could use the combination of an
571
00:25:23,500 --> 00:25:25,416
end-to-end test with Lighthouse.
572
00:25:26,791 --> 00:25:30,666
And you could check that, hey, after like
573
00:25:30,666 --> 00:25:32,166
last week, the startup
574
00:25:32,166 --> 00:25:34,208
time of our app went up 10%.
575
00:25:34,916 --> 00:25:36,875
And they're really thinking, oh, is there
576
00:25:36,875 --> 00:25:38,458
maybe an SDK that we added?
577
00:25:38,458 --> 00:25:39,708
Oh, maybe it's too big.
578
00:25:39,708 --> 00:25:41,166
Maybe we should do, solve
579
00:25:41,166 --> 00:25:42,166
this problem differently.
580
00:25:44,583 --> 00:25:47,125
Without end-to-end tests, it will be
581
00:25:47,125 --> 00:25:49,958
difficult to spot things like that.
582
00:25:50,583 --> 00:25:53,541
We mentioned, we are talking here about
583
00:25:53,541 --> 00:25:58,125
the startup time as time to first
584
00:25:58,125 --> 00:25:59,208
interaction, basically.
585
00:25:59,541 --> 00:25:59,750
Yeah.
586
00:26:01,541 --> 00:26:05,750
Another scenario where I found it
587
00:26:05,750 --> 00:26:11,208
interesting was doing visual,
588
00:26:12,333 --> 00:26:14,958
actually doing something that I said it's
589
00:26:14,958 --> 00:26:16,250
impossible event-to-end test.
590
00:26:16,500 --> 00:26:18,833
So testing for visual glitches.
591
00:26:19,958 --> 00:26:24,541
But you could imagine you're working on
592
00:26:24,541 --> 00:26:25,875
an app in a small team.
593
00:26:25,875 --> 00:26:26,708
You have like a single
594
00:26:26,708 --> 00:26:28,625
designer, you're making a release.
595
00:26:29,041 --> 00:26:31,333
And it's a company
596
00:26:31,333 --> 00:26:33,041
where design is important.
597
00:26:33,250 --> 00:26:34,916
They would like to have like design
598
00:26:34,916 --> 00:26:37,458
approved builds where somebody looks
599
00:26:37,458 --> 00:26:39,208
through them and says it looks fine.
600
00:26:39,916 --> 00:26:41,833
And usually you would like send the build
601
00:26:41,833 --> 00:26:43,958
to the designer who click
602
00:26:43,958 --> 00:26:45,291
through it on their device.
603
00:26:45,291 --> 00:26:47,041
And depending on how much time they have,
604
00:26:47,500 --> 00:26:49,583
they will click a bit more or a bit less,
605
00:26:49,583 --> 00:26:52,083
but still it's one device, one person.
606
00:26:52,291 --> 00:26:53,250
It does not really
607
00:26:53,250 --> 00:26:56,208
seems like a reliable check.
608
00:26:56,208 --> 00:26:58,833
But what you could do is have an
609
00:26:58,833 --> 00:27:01,625
end-to-end test that walks through your
610
00:27:01,625 --> 00:27:03,500
app and captures screenshots
611
00:27:03,500 --> 00:27:05,625
of all the screens on the way.
612
00:27:05,875 --> 00:27:07,875
And you could run the same end-to-end
613
00:27:07,875 --> 00:27:11,333
test on three different sizes of Apple
614
00:27:11,375 --> 00:27:13,750
devices, three different sizes of Android
615
00:27:13,750 --> 00:27:17,416
devices, run it in two different
616
00:27:17,416 --> 00:27:20,833
languages, make screenshots of every
617
00:27:20,833 --> 00:27:23,875
single screen, combine them into an HTML
618
00:27:23,875 --> 00:27:26,458
file, upload it to S3
619
00:27:26,458 --> 00:27:28,250
and send the link to Slack.
620
00:27:28,791 --> 00:27:30,375
And then your designer who needs to
621
00:27:30,375 --> 00:27:33,166
design approve it, they just open a big
622
00:27:33,166 --> 00:27:36,583
HTML file, scroll through it and see if
623
00:27:36,583 --> 00:27:38,000
something seems out of order.
624
00:27:38,833 --> 00:27:40,041
It's so much faster and
625
00:27:40,041 --> 00:27:41,208
so much easier for them
626
00:27:42,333 --> 00:27:43,208
to do this task.
627
00:27:43,583 --> 00:27:45,916
And it's only, and it's internally it's
628
00:27:45,916 --> 00:27:47,750
powered by the end-to-end test suite.
629
00:27:48,125 --> 00:27:51,458
And of course, it's kind of end-to-end
630
00:27:51,458 --> 00:27:53,291
test in a way that...
631
00:27:53,708 --> 00:27:53,916
Smoke test.
632
00:27:54,541 --> 00:27:55,500
Yeah.
633
00:27:56,458 --> 00:27:58,291
If the app completely blows up, like it
634
00:27:58,291 --> 00:28:00,208
doesn't build or it doesn't load or it
635
00:28:00,333 --> 00:28:02,541
crashes during the end-to-end test,
636
00:28:03,208 --> 00:28:04,125
there's something there.
637
00:28:04,125 --> 00:28:06,666
Like you rely on the test to be finished.
638
00:28:06,666 --> 00:28:07,750
It gets to the last screen
639
00:28:07,750 --> 00:28:09,750
and takes all the screenshots.
640
00:28:10,750 --> 00:28:13,750
But I think that the main value there is
641
00:28:13,750 --> 00:28:15,625
the document that it creates, not the
642
00:28:15,625 --> 00:28:19,000
actual run of the end-to-end test suite.
643
00:28:19,000 --> 00:28:21,625
It's a human who's doing the actual test
644
00:28:21,625 --> 00:28:24,208
and the script is
645
00:28:24,208 --> 00:28:25,666
just creating artifacts.
646
00:28:25,666 --> 00:28:26,000
Right.
647
00:28:28,625 --> 00:28:33,583
So yeah, I think those are two scenarios
648
00:28:33,583 --> 00:28:38,166
where I see the value, but
649
00:28:38,166 --> 00:28:43,541
other than that, I don't.
650
00:28:43,875 --> 00:28:48,916
I only see pain and misery, except in
651
00:28:48,916 --> 00:28:51,333
some teams, they managed to figure it out
652
00:28:51,333 --> 00:28:52,916
to minimize the pain.
653
00:28:53,500 --> 00:28:55,083
And there are some strategies how to...
654
00:28:56,208 --> 00:29:00,500
If it's not your call, whether you need
655
00:29:00,500 --> 00:29:03,166
to have end-to-end tests and it comes
656
00:29:03,166 --> 00:29:06,000
from upstairs that it needs to be done
657
00:29:06,000 --> 00:29:08,625
and you want some tips and tricks on how
658
00:29:08,625 --> 00:29:12,708
to make it work relatively pain-free,
659
00:29:14,333 --> 00:29:16,208
then we might have some tips for you.
660
00:29:16,416 --> 00:29:21,375
But in general, I think you should
661
00:29:21,375 --> 00:29:25,625
consider it twice before committing to
662
00:29:25,625 --> 00:29:27,041
maintaining a suite like this.
663
00:29:27,333 --> 00:29:30,416
To be clear, what exactly are you
664
00:29:30,416 --> 00:29:35,333
expecting to get out of this exercise?
665
00:29:36,041 --> 00:29:38,625
Because perhaps it would be much cheaper
666
00:29:38,625 --> 00:29:43,791
and have better results to have an
667
00:29:43,791 --> 00:29:44,125
additional QA person.
668
00:29:44,166 --> 00:29:45,666
Yeah.
669
00:29:46,250 --> 00:29:46,541
Hired.
670
00:29:47,666 --> 00:29:50,208
So yeah, you should run the
671
00:29:50,208 --> 00:29:51,791
numbers, I think in that case.
672
00:29:52,208 --> 00:29:52,416
Yeah.
673
00:29:52,875 --> 00:29:55,125
So for some quick tips of how to
674
00:29:55,125 --> 00:29:56,666
implement them, like
675
00:29:56,666 --> 00:29:58,291
what to pay attention to,
676
00:30:00,041 --> 00:30:04,375
one is optimizing the costs would be...
677
00:30:07,000 --> 00:30:09,166
If you don't want to pay for the cloud
678
00:30:09,166 --> 00:30:12,916
infrastructure, well, one way to optimize
679
00:30:12,916 --> 00:30:14,125
it is to have the computer
680
00:30:14,125 --> 00:30:15,708
in your office that runs it.
681
00:30:16,916 --> 00:30:19,625
Another way is to optimize the time you
682
00:30:19,625 --> 00:30:21,500
spent running on the cloud
683
00:30:21,500 --> 00:30:22,625
infrastructure, which means
684
00:30:22,625 --> 00:30:25,041
you cash as much as you can.
685
00:30:27,083 --> 00:30:30,708
Because well, we're talking about mobile
686
00:30:30,708 --> 00:30:32,750
apps in general, but the whole
687
00:30:32,750 --> 00:30:34,000
podcast is about React Native.
688
00:30:34,208 --> 00:30:39,333
So when working with React Native, it is
689
00:30:39,333 --> 00:30:42,375
most often you just change JavaScript,
690
00:30:42,625 --> 00:30:44,250
you don't touch the native parts.
691
00:30:45,166 --> 00:30:49,333
So 90 something percent of the time, you
692
00:30:49,333 --> 00:30:51,958
can just reuse the native build across
693
00:30:53,125 --> 00:30:53,791
different runs.
694
00:30:54,250 --> 00:30:56,458
So you could just cache it, make sure
695
00:30:56,458 --> 00:30:59,416
that you just modify JavaScript and just
696
00:30:59,416 --> 00:31:01,666
load the new JS bundle on top of that.
697
00:31:02,833 --> 00:31:06,583
This would be, I know, 10 minutes off.
698
00:31:07,041 --> 00:31:10,375
You can as well cache JS bundles if you
699
00:31:10,375 --> 00:31:12,750
just add new tests and so
700
00:31:12,750 --> 00:31:13,833
on, so you don't have to.
701
00:31:14,208 --> 00:31:14,416
True.
702
00:31:15,083 --> 00:31:15,291
True.
703
00:31:16,458 --> 00:31:19,500
So basically caching is a good strategy.
704
00:31:20,000 --> 00:31:20,208
Yeah.
705
00:31:20,333 --> 00:31:24,583
I think also it's very important to spend
706
00:31:24,583 --> 00:31:28,875
the time to learn the tool, because now
707
00:31:28,875 --> 00:31:31,291
we're in the Maestro age, right?
708
00:31:31,458 --> 00:31:34,083
Everybody's saying that, oh, Maestro is
709
00:31:34,083 --> 00:31:36,166
so cool, and detox was so shit, like
710
00:31:36,166 --> 00:31:37,958
we're sweet and we're so much better.
711
00:31:38,916 --> 00:31:41,500
And personally, I don't have much
712
00:31:42,083 --> 00:31:43,166
experience with Maestro.
713
00:31:43,500 --> 00:31:48,125
I implemented a couple of tests back when
714
00:31:48,125 --> 00:31:49,666
it was just like coming out.
715
00:31:50,625 --> 00:31:54,333
And I hit the same issues I was hitting
716
00:31:54,333 --> 00:31:57,666
in detox that like I have elements that
717
00:31:57,666 --> 00:32:00,666
for some reason Maestro doesn't see them.
718
00:32:00,666 --> 00:32:03,708
They're like some APIs that I would like
719
00:32:03,708 --> 00:32:04,458
to use, but they are
720
00:32:04,458 --> 00:32:07,416
not available and so on.
721
00:32:09,208 --> 00:32:12,583
And it was the same between when you were
722
00:32:12,583 --> 00:32:15,083
switching from Appium to detox.
723
00:32:15,750 --> 00:32:19,083
Like detox was definitely a step up, but
724
00:32:19,083 --> 00:32:21,291
it's not a revolution.
725
00:32:21,708 --> 00:32:22,708
It does not really
726
00:32:22,708 --> 00:32:24,333
write the test for you.
727
00:32:24,708 --> 00:32:26,375
You still have to maintain it.
728
00:32:27,166 --> 00:32:30,291
And I think the investment of learning
729
00:32:30,291 --> 00:32:32,791
the tool is the best
730
00:32:32,791 --> 00:32:35,083
thing you can do early on.
731
00:32:35,958 --> 00:32:40,416
The only teams that I know that they are
732
00:32:40,416 --> 00:32:42,666
happy with their detox setup, they see no
733
00:32:42,666 --> 00:32:45,625
problem with their end to end suite are
734
00:32:45,625 --> 00:32:47,416
the teams that sat down and basically
735
00:32:47,416 --> 00:32:49,958
read through the detox source code.
736
00:32:50,625 --> 00:32:52,541
And they know exactly what's going on.
737
00:32:52,541 --> 00:32:54,458
And they optimized the way they write
738
00:32:54,458 --> 00:32:58,791
tests for not even for the documentation
739
00:32:58,791 --> 00:33:00,916
of detox, filing all the best practices,
740
00:33:00,916 --> 00:33:01,916
but for the actual
741
00:33:01,916 --> 00:33:05,500
implementation, what's done underneath.
742
00:33:06,666 --> 00:33:10,625
And I think in this particular case, it's
743
00:33:10,625 --> 00:33:14,125
crucial that you know deeply the tool
744
00:33:14,125 --> 00:33:16,250
that you're using, be it Maestro or
745
00:33:16,250 --> 00:33:18,041
detox, if you want to
746
00:33:18,041 --> 00:33:18,958
avoid the flakiness.
747
00:33:20,250 --> 00:33:22,041
Because every situation where a test
748
00:33:22,041 --> 00:33:23,708
fails and you don't know why it fails,
749
00:33:24,083 --> 00:33:26,250
that's a warning light.
750
00:33:26,541 --> 00:33:28,416
That's probably there's some knowledge
751
00:33:29,208 --> 00:33:31,791
that you could have, but you don't.
752
00:33:32,583 --> 00:33:34,875
And it would be useful to learn it.
753
00:33:38,166 --> 00:33:40,500
Even if it just cost me grace hitting
754
00:33:40,500 --> 00:33:43,666
your server, you should know for sure
755
00:33:43,666 --> 00:33:45,291
whether this will cost me grace or
756
00:33:45,291 --> 00:33:46,750
something, something
757
00:33:46,750 --> 00:33:48,166
you could have prevented.
758
00:33:50,625 --> 00:33:54,958
Those are the two tips I would have.
759
00:33:55,000 --> 00:33:56,000
Awesome.
760
00:33:56,458 --> 00:33:58,916
Anything to add Patrick?
761
00:34:00,041 --> 00:34:03,000
No, I think we covered the most
762
00:34:03,000 --> 00:34:07,458
interesting things about end to end tests
763
00:34:07,458 --> 00:34:09,708
and their quirks and features.
764
00:34:11,083 --> 00:34:13,958
Yeah, I think that would be it for today.
765
00:34:14,291 --> 00:34:14,541
Awesome.
766
00:34:15,083 --> 00:34:17,791
Bottom line, friends don't let friends
767
00:34:17,791 --> 00:34:19,000
write end to end tests.
768
00:34:20,000 --> 00:34:22,000
Unless they have a good reason.
769
00:34:22,416 --> 00:34:22,583
Nice.
770
00:34:22,791 --> 00:34:23,000
Cool.
771
00:34:23,500 --> 00:34:24,750
Thank you, Patrick, for
772
00:34:24,750 --> 00:34:26,750
this opportunity to run.
773
00:34:27,041 --> 00:34:27,500
Thank you.
774
00:34:28,208 --> 00:34:30,916
I hope that's informational and I hope
775
00:34:30,916 --> 00:34:31,833
that's something that you
776
00:34:31,833 --> 00:34:34,041
can send to your manager.
777
00:34:34,875 --> 00:34:36,791
Well, our listeners can send to their
778
00:34:36,791 --> 00:34:39,500
managers when they casually ask, "How did
779
00:34:39,500 --> 00:34:40,708
you consider adding
780
00:34:40,708 --> 00:34:41,583
some end to end tests?"
781
00:34:42,000 --> 00:34:45,666
We've been experiencing bugs lately.
782
00:34:45,708 --> 00:34:47,166
Maybe we should do end to end tests.
783
00:34:47,666 --> 00:34:51,583
For this manager, we can just say that
784
00:34:51,583 --> 00:34:53,333
end to end tests are
785
00:34:53,333 --> 00:34:54,500
not the solution here.
786
00:34:54,500 --> 00:34:55,500
The solution here is
787
00:34:55,500 --> 00:34:56,416
to write correct code.
788
00:34:56,833 --> 00:34:57,208
Nice.
789
00:35:00,041 --> 00:35:02,458
Actually, that's a good segue because if
790
00:35:02,458 --> 00:35:05,375
you are interested in how to write a good
791
00:35:05,375 --> 00:35:09,666
code, then maybe you should subscribe to
792
00:35:09,666 --> 00:35:10,708
Wojciech Architecture
793
00:35:10,708 --> 00:35:13,083
course for React Native.
794
00:35:13,541 --> 00:35:17,375
Yeah, I would be happy
795
00:35:17,375 --> 00:35:18,625
to welcome you there.
796
00:35:19,416 --> 00:35:23,000
It's on brainsandbeards.com slash mrn,
797
00:35:23,458 --> 00:35:25,166
like maintainable React Native.
798
00:35:26,458 --> 00:35:29,083
And it's the course where we try to
799
00:35:29,083 --> 00:35:31,333
explain how to write React Native apps
800
00:35:31,333 --> 00:35:33,583
without going crazy.
801
00:35:33,583 --> 00:35:33,791
Cool.
802
00:35:35,083 --> 00:35:35,291
Okay.
803
00:35:35,583 --> 00:35:39,291
Thank you all for listening and I hope to
804
00:35:39,291 --> 00:35:40,958
see you soon in the next podcast.
805
00:36:07,375 --> 00:36:07,541
Yeah. Ciao.
806
00:36:07,750 --> 00:36:09,041
See you soon.
807
00:36:09,041 --> 00:36:09,333
Take care.
808
00:36:10,083 --> 00:36:12,125
Bye. Bye.