Running and monitoring a low maintenance Java web application

Running and monitoring a low maintenance web application Looking into the ElasticCC platform Alexander Reelsen alex@elastic.co | @spinscale

Today’s goal How to run and maintain a modern web java application with minimal resources

Agenda Conference app as a platform? Build vs. Buy? Requirements Technologies used Persistence layer Downtime free rollouts Monitoring Frontend for backend devs Dashboards for tracking

Requirements Schedule & Speaker info Registration Feedback & Rating Quiz

Schedule Display Filtering Personalization Provide ical file

Registration No password data in this system! SAML: Use Elastic Cloud login Danger: Breaks single page flow!

Feedback Rating Feedback wall

Build vs. Buy Customization SAML integration Hefty price tag Focus on commercial events, not free ones Use your own technologies in production! — Me

Requirements Speed (rendering) Ease of use & feature development Fast startup Monitoring/APM Docker image to deploy to k8s Low maintenance

Technologies used Javalin (Jetty) JTE pac4j Elastic Stack Config file parsing: typesafe/lightbend config Testing: assertJ, testcontainers, equalsverifier Frontend: htmx & hyperscript

Javalin Based on jetty, servlet based No reflection, no annotations Fast startup Extensible Built-in support for WS, SSE, OpenAPI

JTE Fast, typed, built-in escaping Hot reloading in development Build time compilation for production Tag support @import my.Entry @param Entry entry @param boolean verbose = false <h2>${entry.title}</h2> @if(verbose) <h3>${entry.subtitle}</h3> @endif

pac4j The Java security engine to protect all your web applications and web services Authorization Authentication (OAuth, SAML, CAS, OpenID Connect, JWT, SPNEGO) Supports almost 20 web frameworks

Persistence layer CfP process is done via sessionize Contains schedule data, speaker data Allows to add custom fields Has an API, read-only Schedule can be polled and updated every 10 minutes No need for persistence for the schedule, but…

Persistence layer Storing registrations, ratings, feedback, custom agenda Elasticsearch Use new Elasticsearch Java Client to provide feedback

Elasticsearch persistence public record Rating(String sessionId, String userId, Integer rating, List<Feedback> feedbacks, OffsetDateTime createdAt) { } public class RatingService { public static final String INDEX = “elasticcc-ratings”; private final ElasticsearchClient client; public RatingService(ElasticsearchClient client) { this.client = client; } } public void save(Rating rating) throws IOException { client.index(b -> b.index(INDEX) .id(rating.sessionId() + “-” + rating.userId()) .document(rating) ); }

Elasticsearch persistence public final static ObjectMapper mapper = new ObjectMapper() .registerModule(new JavaTimeModule()) .setPropertyNamingStrategy(PropertyNamingStrategies.SNAKE_CASE) .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false) .setSerializationInclusion(JsonInclude.Include.NON_NULL); public ElasticsearchClient toElasticsearchClient(Config config) { final RestClientBuilder builder = RestClient.builder(HttpHost.create(config.getString(“host”))); if (config.hasPath(“user”) && config.hasPath(“password”)) { final String user = config.getString(“user”); final String password = config.getString(“password”); final CredentialsProvider credentialsProvider = new BasicCredentialsProvider(); credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password)); builder.setHttpClientConfigCallback(b -> b.setDefaultCredentialsProvider(credentialsProvider)); } RestClient restClient = builder.build(); ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper(mapper)); return new ElasticsearchClient(transport); }

Elasticsearch persistence public Map<String, QuizAnswer> loadAnswers(String userId, List<String> questionIds) throws IOException { List<FieldValue> fieldValues = questionIds.stream().map(id -> FieldValue.of(x -> x.stringValue(id))).toList(); SearchResponse<QuizAnswer> response = client.search(b -> b .index(ANSWER_INDEX) .size(15) .query(q -> q.bool(bq -> bq .filter(fb -> fb .term(t -> t.field(“user_id.keyword”).value(FieldValue.of(userId)))) .filter(fb -> fb .terms(t -> t.field(“question_id.keyword”).terms(TermsQueryField.of(tqb -> tqb.value(fieldValues))))) )) , QuizAnswer.class); return response.hits().hits().stream().collect(Collectors.toMap(h -> h.source().questionId(), Hit::source)); }

Downtime free rollouts kubectl rollout restart deployment/elasticcc-app readiness probes liveness probes persistent sessions (> 1 replicas, round robin)

Persistent sessions public static SessionHandler createSessionHandler(ElasticsearchClient client) { SessionHandler sessionHandler = new SessionHandler(); sessionHandler.setMaxInactiveInterval(86_400 * 7); // full week sessionHandler.setHttpOnly(true); sessionHandler.setSameSite(HttpCookie.SameSite.STRICT); sessionHandler.setSecureRequestOnly(true); } SessionCache sessionCache = new NullSessionCache(sessionHandler); sessionCache.setSaveOnCreate(true); sessionCache.setFlushOnResponseCommit(true); sessionCache.setSessionDataStore(new ElasticsearchSessionDataStore(client)); sessionHandler.setSessionCache(sessionCache); return sessionHandler;

Persistent sessions @Override public void doStore(String id, SessionData data, long lastSaveTime) throws Exception { if (!isSessionIdValid(id)) { return; } final String base64Session; try (ByteArrayOutputStream bos = new ByteArrayOutputStream(); ObjectOutputStream oos = new ObjectOutputStream(bos)) { SessionData.serializeAttributes(data, oos); oos.flush(); base64Session = Base64.getEncoder().encodeToString(bos.toByteArray()); } ElasticsearchSession session = new ElasticsearchSession(id, _context.getCanonicalContextPath(), _context.getVhost(), data.getLastNode(), data.getCreated(), data.getAccessed(), data.getLastAccessed(), data.getCookieSet(), data.getExpiry(), data.getMaxInactiveMs(), base64Session); } client.index(b -> b.index(INDEX).id(id).document(session));

Persistent sessions - problem: bots About 99% of the requests are bots, security scanners, that create a new session per request Persists a six digit amount of sessions per day Solution: Persist session only, if users are logged in

Persistent sessions - login check app.after(“/login”, ctx -> { ctx.sessionAttribute(“persist_session”, true); }); public void doStore(String id, SessionData data, long lastSaveTime) throws Exception { // validation // check if should be persisted Boolean persistSession = (Boolean) data.getAttribute(“persist_session”); if (persistSession == null || persistSession == false) { return; } // persist // … }

Persistent sessions - login check much less processing time for most requests much less Elasticsearch indexing work less CPU time wasted, less useless work done

Monitoring Elastic APM, from the browser to the backend

Elastic APM setup // set version at build time to annotate rollouts setElasticApmServiceVersion(); System.setProperty(“elastic.apm.environment”, isProduction() ? “prod” : “dev”); System.setProperty(“elastic.apm.application_packages”, “co.elastic.community”); // add a span for rendering .jte templates and a span for Context.render() System.setProperty(“elastic.apm.trace_methods”, “io.javalin.plugin.rendering.template.JavalinJte#render,io.javalin.http.Context#render”); // those should not be shown in the APM UI, as they are controlling flow/logic System.setProperty(“elastic.apm.ignore_exceptions”, “io.javalin.http.RedirectResponse,*ResponseNoopException”); // this is to have consistent JVM stats across restarts in the APM UI System.setProperty(“elastic.apm.service_node_name”, “elasticcc-service-java”); // programmatic attachment, no need to configure an agent ElasticApmAttacher.attach();

Elastic APM setup @param co.elastic.apm.api.Transaction transaction <script> ;(function(d, s, c) { var j = d.createElement(s), t = d.getElementsByTagName(s)[0] j.src = ‘https://unpkg.com/@elastic/apm-rum@5.10.0/dist/bundles/elastic-apm-rum.umd.min.js’ j.onload = function() {elasticApm.init(c)} t.parentNode.insertBefore(j, t) })(document, ‘script’, { serviceName: ‘elasticcc-app-frontend’, environment: ‘prod’, serverUrl: ‘https://elasticcc-app.apm.us-central1.gcp.cloud.es.io’, pageLoadTraceId: “${transaction.getTraceId()}”, pageLoadSpanId: “${transaction.ensureParentId()}”, pageLoadSampled: ${transaction.isSampled()} }) </script> </body> </html>

Monitoring - Latency

Monitoring - Transactions

Errors NPE detection

JVM monitoring ZGC: CPU: Mem:

Frontend …is part of your performance!

Frontend for backend devs SPAs as a no-go due to lacking skills htmx for making websites dynamic, but rendering server side hyperscript for events

htmx alternatives: Hotwire, Turbo HTML tag enrichment to send AJAX request render results server side configure HTML element to be updated with the response optional: update browser history

htmx @param String sessionId <ul id=”rating_${sessionId}” style=”list-style-type:none;display:inline;margin-left:0px;”> @for(int i = 1; i <= 5; ++i) <li style=”display:inline;”> <a hx-post=”/session/${sessionId}/rating/${i}” hx-target=”#rating_${sessionId}” hx-swap=”outerHTML” style=”cursor:pointer;text-decoration:none;”> </a> </li> @endfor </ul>

hyperscript hyperscript makes writing event handlers and highly responsive user interfaces easy with a clear, DOM-oriented syntax and by transparently handling asynchronous behavior for you — easier than callbacks, promises, even async/await.

hyperscript <div style=”display: grid; grid-template-columns:1fr 1fr 1fr 1fr;”> @for(var feedback : Feedback.values()) !{var hasFeedback = rating != null && rating.feedbacks() != null && rating.feedbacks().contains(feedback);} <div class=”@if(hasFeedback)button-primary-blue@elsebutton-secondary-blue@endif” _=”on click toggle .button-secondary-blue toggle .button-primary-blue toggle [@checked=true] on #${feedback.name().toLowerCase(Locale.ROOT)}” >${feedback.displayName()} </div> <input type=”checkbox” style=”display:none” id=”${feedback.name().toLowerCase(Locale.ROOT)}” name=”${feedback.name()}” checked=”${hasFeedback}”/> @endfor </div>

Registrations

HTTP requests

Docker build multi layer build template precompilation use gradle application plugin to create shell scripts to start apps configure JVM options in build.gradle , i.e. ZGC usage

Docker build FROM gradle:7.4-jdk17 AS build COPY —chown=gradle:gradle . /home/gradle/src WORKDIR /home/gradle/src RUN gradle clean —no-daemon RUN gradle copyDepsToLib —no-daemon RUN gradle check —no-daemon RUN gradle precompileJte —no-daemon RUN gradle copyStartScriptsToBin —no-daemon RUN gradle copyAppToLib —no-daemon ############### FROM —platform=linux/amd64 openjdk:17.0.2-slim-bullseye RUN addgroup —system elasticcc && adduser —system elasticcc —ingroup elasticcc USER elasticcc:elasticcc WORKDIR /elasticcc-app # copy resources COPY … EXPOSE 8080 ENTRYPOINT [ “/elasticcc-app/install/elasticcc-app/bin/elasticcc-app” ]

Testing Unit tests & integration tests Equalsverifier , assertJ , mockito Testcontainers

Testing - Equalsverifier public class EqualsVerifierTests { @Test public void testEqualsVerifier() { EqualsVerifier.forClass(User.class) .suppress(Warning.STRICT_INHERITANCE, Warning.NONFINAL_FIELDS) .verify(); } } EqualsVerifier.forClass(Rating.class).verify(); EqualsVerifier.forClass(QuizAnswer.class).verify(); EqualsVerifier.forClass(QuizQuestion.class).verify(); EqualsVerifier.forClass(QuizQuestionAnswer.class).verify();

Testing - Controller @Test public void testShowSessionWhenNotLoggedIn() throws Exception { when(ctx.pathParam(eq(“id”))).thenReturn(“123”); final Session session = createSession(“123”); when(sessionService.session(eq(session.id()))).thenReturn(session); // not logged in when(ctx.sessionAttribute(eq(“user”))).thenReturn(null); controller.showSession().handle(ctx); assertThat(ctx).hasRenderTemplate(“session”); assertThat(ctx).renderMap().containsEntry(“session”, session); verifyNoInteractions(quizService); verifyNoInteractions(ratingService); }

Testing - Testcontainers @Tag(“slow”) public class ElasticsearchSessionDataStoreTests extends AbstractElasticsearchContainerTest { @Test public void testStoreAndSerializeUserEntity() throws Exception { final long millis = ZonedDateTime.of(2021, 11, 8, 12, 34, 56, 123_000_000, ZoneOffset.UTC) .toInstant() .toEpochMilli(); SessionData sessionData = new SessionData(sessionId, “”, “0.0.0.0”, millis, millis + 1000, millis + 1000, 1000); User user = new User(); user.setRegion(User.Region.EMEA); user.setName(“Paul Paulson”); sessionData.putAllAttributes(Map.of(“user”, user, “persist_session”, true)); sessionStore.store(sessionData.getId(), sessionData); } } final SessionData loadedSession = sessionStore.load(sessionData.getId()); assertThat(loadedSession.getAllAttributes()).containsEntry(“user”, user);

Keep in mind… low maintenance != no maintenance dependabot security is an ongoing task

Going live Secrets via vault JSON logging Access to logfiles k8s readiness/liveness probes

Summary

Summary Modern Minimal resources Other java developers commit changes Low maintenance Maximum performance? Graal

Resources Community Conference website Web server: Javalin, Jetty, JTE, pac4j Elastic: Elastic Stack, Elasticsearch Java Client, Elastic Java APM Agent Configuration: typesafe config library Testing: assertJ, mockito, Testcontainers, EqualsVerifier Frontend: htmx, hyperscript, hotwire

Thanks for listening Q&A Alexander Reelsen alex@elastic.co | @spinscale