In this comprehensive guide, the author walks readers through building a Flight Search Application using Elasticsearch and Kibana, leveraging a preloaded sample flight dataset in Kibana. The article assumes a basic understanding of Elasticsearch and Kibana, and it is recommended to read the author's previous articles on these topics for a fuller context. Setting Up the Environment To begin, the author provides a docker-compose file to set up Elasticsearch and Kibana locally. This file specifies the images for both services, their configurations, and network settings. Running this file ensures Elasticsearch and Kibana are up and running, and accessible via the specified ports. ```yaml version: '3.8' services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.7.1 expose: - 9200 environment: - xpack.security.enabled=false - "discovery.type=single-node" - ELASTIC_USERNAME=elastic - ELASTIC_PASSWORD=DkIedPPSCb networks: - es-net ports: - 9200:9200 volumes: - elasticsearch-data:/usr/share/elasticsearch/data kibana: image: docker.elastic.co/kibana/kibana:8.7.1 environment: - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 expose: - 5601 networks: - es-net depends_on: - elasticsearch ports: - 3036:5601 volumes: - kibana-data:/usr/share/kibana/data networks: es-net: driver: bridge volumes: elasticsearch-data: driver: local kibana-data: driver: local ``` After setting up the environment, users are directed to the Kibana home page to install the sample flight dataset. Building the Application Models The core model in this application is the Flight class, which maps the fields from the Kibana sample dataset. These fields include essential details such as flight number, carrier, origin and destination city names, countries, timestamps, delays, cancellations, average ticket price, and weather conditions. csharp public class Flight { [JsonPropertyName("FlightNum")] public string FlightNumber { get; set; } [JsonPropertyName("Carrier")] public string Carrier { get; set; } [JsonPropertyName("OriginCityName")] public string OriginCityName { get; set; } [JsonPropertyName("OriginCountry")] public string OriginCountry { get; set; } [JsonPropertyName("DestCityName")] public string DestCityName { get; set; } [JsonPropertyName("DestCountry")] public string DestCountry { get; set; } [JsonPropertyName("timestamp")] public DateTime Timestamp { get; set; } [JsonPropertyName("FlightDelay")] public bool FlightDelay { get; set; } [JsonPropertyName("Cancelled")] public bool FlightCancelled { get; set; } [JsonPropertyName("FlightDelayMin")] public int FlightDelayMin { get; set; } [JsonPropertyName("FlightTimeMin")] public float FlightTimeMin { get; set; } [JsonPropertyName("DistanceKilometers")] public float DistanceKilometers { get; set; } [JsonPropertyName("DistanceMiles")] public float DistanceMiles { get; set; } [JsonPropertyName("OriginWeather")] public string OriginWeather { get; set; } [JsonPropertyName("AvgTicketPrice")] public float AvgTicketPrice { get; set; } [JsonPropertyName("OriginLocation")] public GeoLocation OriginLocation { get; set; } [JsonPropertyName("DestLocation")] public GeoLocation DestLocation { get; set; } } DTOs Two DTOs (Data Transfer Objects) are defined to facilitate data handling and search operations: FlightDto: Simplifies the Flight model for user interface representation. FlightSearchDto: Represents the searchable and filterable fields. ```csharp public class FlightDto { public string FlightNumber { get; set; } public string Carrier { get; set; } public string OriginCityName { get; set; } public string OriginCountry { get; set; } public string DestCityName { get; set; } public string DestCountry { get; set; } public DateTime Timestamp { get; set; } public bool FlightDelay { get; set; } public int FlightDelayMin { get; set; } public bool FlightCancelled { get; set; } public float FlightTimeMin { get; set; } public double DistanceKilometers { get; set; } public string OriginWeather { get; set; } public double AvgTicketPrice { get; set; } } public class FlightSearchDto { public string OriginCityName { get; set; } public string OriginCountry { get; set; } public string DestCityName { get; set; } public string DestCountry { get; set; } public double? AvgTicketPrice { get; set; } public DateTime? FromDate { get; set; } public DateTime? ToDate { get; set; } public bool? FlightDelay { get; set; } public bool? FlightCancelled { get; set; } public string Carrier { get; set; } } ``` Flight Repository The FlightRepository class handles the interaction with Elasticsearch. It includes a method, SearchAsync, which constructs Elasticsearch queries based on the provided search criteria. The method checks each field for a non-null value and adds corresponding match, range, or term queries to the query list. If no criteria are provided, it returns all flight data. The CalculateResultSet method then executes the query, retrieves the results, and handles pagination. ```csharp public class FlightRepository { private readonly ElasticsearchClient _elasticsearchClient; private string IndexName = "kibana_sample_data_flights"; public FlightRepository(ElasticsearchClient elasticsearchClient) { _elasticsearchClient = elasticsearchClient; } public async Task<(List<Flight> list, long count)> SearchAsync(FlightSearchDto flight, int page, int pageSize) { var queryList = new List<Action<QueryDescriptor<Flight>>>(); if (flight is null) { queryList.Add(q => q.MatchAll(m => m.QueryName("MatchAll"))); return await CalculateResultSet(queryList, page, pageSize); } if (!string.IsNullOrEmpty(flight.OriginCityName)) { queryList.Add(q => q.Match(m => m.Field(f => f.OriginCityName).Query(flight.OriginCityName))); } if (!string.IsNullOrEmpty(flight.OriginCountry)) { queryList.Add(q => q.Match(m => m.Field(f => f.OriginCountry).Query(flight.OriginCountry))); } if (!string.IsNullOrEmpty(flight.DestCityName)) { queryList.Add(q => q.Match(m => m.Field(f => f.DestCityName).Query(flight.DestCityName))); } if (!string.IsNullOrEmpty(flight.DestCountry)) { queryList.Add(q => q.Match(m => m.Field(f => f.DestCountry).Query(flight.DestCountry))); } if (flight.AvgTicketPrice.HasValue) { queryList.Add(q => q.Range(r => r.NumberRange(n => n.Field(f => f.AvgTicketPrice).Gte(flight.MinTicketPrice).Lte(flight.MaxTicketPrice)))); } if (flight.FlightDelay.HasValue) { queryList.Add(q => q.Term(t => t.Field(f => f.FlightDelay).Value(flight.FlightDelay.Value))); } if (!queryList.Any()) { queryList.Add(q => q.MatchAll(m => m.QueryName("MatchAllQuery"))); } return await CalculateResultSet(queryList, page, pageSize); } private async Task<(List<Flight> list, long count)> CalculateResultSet(List<Action<QueryDescriptor<Flight>>> queryList, int page, int pageSize) { var result = await _elasticsearchClient.SearchAsync<Flight>(s => s .Index(IndexName) .Size(pageSize) .From((page - 1) * pageSize) .Query(q => q.Bool(b => b.Must(queryList.ToArray()))) ); if (result.IsValidResponse && result.Hits != null) { var documents = result.Hits.Select(hit => hit.Source).ToList(); return (documents, result.Total); } else { return (new List<Flight>(), 0); // Return empty list on error } } } ``` Flight Service The FlightService class acts as an intermediary between the repository and the controller. It calls the SearchAsync method from the FlightRepository and maps the results to FlightDto objects. It also calculates the total number of pages for pagination. ```csharp public class FlightService { private readonly FlightRepository _flightRepository; public FlightService(FlightRepository flightRepository) { _flightRepository = flightRepository; } public async Task<(List<FlightDto> list, long totalCount, long pageLinkCount)> SearchAsync(FlightSearchDto flight, int page, int pageSize) { var (flightList, totalCount) = await _flightRepository.SearchAsync(flight, page, pageSize); long pageLinkCount = (totalCount + pageSize - 1) / pageSize; var flightListDto = flightList.Select(x => new FlightDto { AvgTicketPrice = x.AvgTicketPrice, Carrier = x.Carrier, DestCityName = x.DestCityName, DestCountry = x.DestCountry, DistanceKilometers = x.DistanceKilometers, FlightCancelled = x.FlightCancelled, FlightDelay = x.FlightDelay, FlightDelayMin = x.FlightDelayMin, FlightNumber = x.FlightNumber, FlightTimeMin = x.FlightTimeMin, OriginCityName = x.OriginCityName, OriginCountry = x.OriginCountry, OriginWeather = x.OriginWeather, Timestamp = x.Timestamp }).ToList(); return (flightListDto, totalCount, pageLinkCount); } } ``` Controller The FlightController class exposes the search functionality via an API endpoint. It accepts a SearchDto object from the user, which includes the search criteria, current page, and page size. The SearchAsync method processes the request and returns the filtered results. ```csharp [Route("api/[controller]")] [ApiController] public class FlightController : ControllerBase { private readonly FlightService _flightService; public FlightController(FlightService flightService) { _flightService = flightService; } [HttpPost("search")] public async Task<IActionResult> SearchAsync([FromQuery] SearchDto searchDto) { var (flightList, totalCount, pageLinkCount) = await _flightService.SearchAsync(searchDto.FlightSearchDto, searchDto.Page, searchDto.PageSize); searchDto.List = flightList.ToList(); searchDto.TotalCount = totalCount; searchDto.PageLinkCount = pageLinkCount; return Ok(searchDto); } } ``` Program.cs Configuration The Program.cs file configures the ASP.NET Core application, including registering the Elasticsearch client, controllers, and other necessary services. ```csharp var builder = WebApplication.CreateBuilder(args); builder.Services.AddControllers().AddJsonOptions(options => options.JsonSerializerOptions.DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull); builder.Services.AddEndpointsApiExplorer(); builder.Services.AddElastic(); builder.Services.AddSwaggerGen(); builder.Services.AddScoped(); builder.Services.AddScoped(); var app = builder.Build(); if (app.Environment.IsDevelopment()) { app.UseSwagger(); app.UseSwaggerUI(); } app.UseHttpsRedirection(); app.UseAuthorization(); app.MapControllers(); app.Run(); ``` Testing the Application The author provides a few test scenarios to validate the application's functionality: Basic Search: Searching flights originating from Rome. Multiple Criteria Search: Filtering flights by origin city, destination city, and delay status. Range Search: Searching flights based on origin city, destination city, and ticket price range. Industry Insights and Company Profile Industry insiders praise the use of Elasticsearch for its powerful search and analytics capabilities, particularly in handling large datasets efficiently. The combination of Elasticsearch and Kibana offers a robust solution for real-time data visualization and exploration. The author, Tuğrulhan Karslı, is a seasoned software developer with a strong focus on backend development and data management. His LinkedIn profile can be found at Tuğrulhan Karslı. This step-by-step guide not only helps beginners understand the basics of integrating Elasticsearch into their applications but also provides valuable insights for experienced developers looking to optimize their data handling and search functionalities. The source code for the project is available on GitHub at ElasticsearchFlightApp.