s3mini: Ultra-Lightweight S3 Client for Node and Edge Computing Platforms

s3mini: A Lightweight S3 Client for Modern Tech Stacks In the rapidly evolving landscape of cloud storage and edge computing, developers often face challenges in selecting tools that balance performance with minimal resource usage. One such tool is s3mini, an ultra-lightweight Typescript client designed for S3-compatible object storage. Released in early 2024, s3mini aims to be a versatile solution, working seamlessly across diverse platforms including Node.js, Bun, and Cloudflare Workers, with a focus on edge computing. However, it explicitly lacks browser support due to its reliance on Node.js APIs. Key Features of s3mini Light and Fast: The standout feature of s3mini is its compact size—approximately 14 KB minified—which contributes to its impressive performance, averaging about 15% more operations per second (ops/s) compared to similar clients. This makes it an excellent choice for applications where resource efficiency is paramount. Zero Dependencies: Unlike many S3 clients that come with a suite of dependencies, s3mini stands out by having no external dependencies. This not only reduces the overall package size but also simplifies the installation and deployment process. It supports AWS Signature Version 4 (SigV4) for secure communication, eliminating the need for pre-signed URL requests. Edge Computing Ready: The design of s3mini is particularly suited for edge computing environments, such as Cloudflare Workers. Edge computing requires software to be as lightweight and efficient as possible, and s3mini meets these requirements by offering a pared-down set of essential APIs. These include methods for listing, putting, getting, and deleting objects, as well as advanced features like multipart uploads and downloads. BYOS3 (Bring Your Own S3): s3mini is not limited to just one S3 provider. It has been rigorously tested on several platforms, including Cloudflare R2, Backblaze B2, DigitalOcean Spaces, MinIO, and Garage. This flexibility allows developers to integrate it seamlessly with their preferred storage solutions. Future versions are expected to support additional providers like Ceph and Amazon S3. Performance Testing Initial performance tests were conducted using a local Minio instance. While these results provide a baseline, actual performance may vary depending on the environment and network conditions. Developers are encouraged to conduct their own tests to ensure optimal performance in their specific scenarios. Supported Operations Bucket Operations: - Bucket Exists: Checks if a specified bucket exists. - Create Bucket: Creates a new bucket if it does not exist. - List Buckets: Lists all available buckets. Object Operations: - Object Exists: Verifies if an object is present in a bucket. - Put Object: Uploads an object to the bucket. - Get Object: Retrieves an object from the bucket. - Delete Object: Removes an object from the bucket. - List Objects: Enumerates all objects in a bucket, supporting pagination through the maxKeys option. - Multipart Upload: Handles large files by breaking them into smaller parts and uploading each part independently. - Multipart Download: Allows downloading large files in chunks, enhancing efficiency and reducing memory usage. Installation and Usage To use s3mini, developers can install it via npm: sh npm install s3mini Here is a basic example of how to use s3mini: ```typescript import { s3mini, sanitizeETag } from 's3mini'; const s3client = new s3mini({ accessKeyId: config.accessKeyId, secretAccessKey: config.secretAccessKey, endpoint: config.endpoint, region: config.region, }); // Check if the bucket exists let exists: boolean = false; try { exists = await s3client.bucketExists(); } catch (err) { throw new Error(Failed bucketExists() call, wrong credentials maybe: ${err.message}); } if (!exists) { await s3client.createBucket(); } // Upload an object const smallObjectKey: string = 'small-object.txt'; const smallObjectContent: string = 'Hello, world!'; const objectExists: boolean = await s3client.objectExists(smallObjectKey); if (!objectExists) { const resp: Response = await s3client.putObject(smallObjectKey, smallObjectContent); const etag = sanitizeETag(resp.headers.get('etag')); console.log('ETag:', etag); } // Retrieve an object const objectData: string | null = await s3client.getObject(smallObjectKey); console.log('Object data:', objectData); // List objects in a bucket const list: object[] | null = await s3client.listObjects(); if (list) { console.log('List of objects:', list); } else { console.log('No objects found.'); } // Delete an object const wasDeleted: boolean = await s3client.deleteObject(smallObjectKey); console.log('Object deleted?', wasDeleted); // Multipart Upload const multipartKey = 'multipart-object.txt'; const largeBuffer = new Uint8Array(1024 * 1024 * 15); // 15 MB buffer const partSize = 8 * 1024 * 1024; // 8 MB const totalParts = Math.ceil(largeBuffer.length / partSize); const uploadId = await s3client.getMultipartUploadId(multipartKey); const uploadPromises = []; for (let i = 0; i < totalParts; i++) { const partBuffer = largeBuffer.subarray(i * partSize, (i + 1) * partSize); uploadPromises.push(s3client.uploadPart(multipartKey, uploadId, partBuffer, i + 1)); } const uploadResponses = await Promise.all(uploadPromises); const parts = uploadResponses.map((response, index) => ({ partNumber: index + 1, etag: response.etag, })); const completeResponse = await s3client.completeMultipartUpload(multipartKey, uploadId, parts); const completeEtag = completeResponse.etag; console.log('Complete ETag:', completeEtag); // Multipart Download const rangeStart = 2048 * 1024; // 2 MB const rangeEnd = 8 * 1024 * 1024 * 2; // 16 MB const rangeResponse = await s3client.getObjectRaw(multipartKey, false, rangeStart, rangeEnd); const rangeData = await rangeResponse.arrayBuffer(); console.log('Range data length:', rangeData.byteLength); ``` Security and Community Contributions s3mini is committed to maintaining high security standards and welcomes contributions from the community. If you encounter issues or have feature requests, report them on GitHub, providing as much detail as possible. Pull requests are also encouraged, with a particular emphasis on maintaining the library's lightweight philosophy. When contributing, avoid introducing heavy dependencies and ensure that any new features provide significant value. All contributors are expected to adhere to the code of conduct, fostering a respectful and collaborative environment. Licensing s3mini is released under the MIT License, allowing developers to use, modify, and distribute the software freely. This permissive license makes it an attractive option for both commercial and open-source projects. Sponsorship The continued development and maintenance of s3mini rely on community support. If you find the library valuable, consider sponsoring the project. Your contribution helps ensure ongoing improvements and support for this essential tool. Industry Insights and Evaluation Industry insiders have praised s3mini for its efficiency and flexibility. The zero-dependency approach and compatibility with multiple S3 providers make it a robust choice for developers working in resource-constrained environments. Additionally, its performance benchmarks suggest that it can handle high volumes of operations effectively, making it suitable for large-scale applications. Company Profile: s3mini is maintained by a dedicated group of developers committed to providing high-quality, lightweight tools for modern web applications. Their focus on community involvement and open-source principles has attracted a growing user base and positive feedback from various sectors of the tech community. This library’s minimalist design philosophy aligns well with the growing trend towards lean and efficient software, particularly in the age of edge computing and serverless architectures. As cloud storage and distributed systems become more prevalent, tools like s3mini will play a crucial role in optimizing performance and resource utilization.