Finding and recovering
$310/month in AWS “hidden fixed costs”3 months of rethinking ALB / NAT / Fargate

1. Two phases of review ─ From "feeling like we were on it" to "found it, got it back"

In parallel with the above, we also trimmed the unnecessary metrics the CloudWatch Agent was publishing on its default settings (tactic #5).

From the next section on, we dig into each of these tactics one at a time from three angles: "how much we were paying / why it was wasteful / how we got it back". If you'd rather read by phase, use the links above to jump to the corresponding tactic.

2. Consolidating 3 ALBs into 1 (23 listener rules)

What we did

We consolidated the per-service ALBs from 3 down to 1. Without changing the number of domains, we created 23 listener rules on the HTTPS listener (443) that route traffic to the right target group based on the host header (Host: example.com) + path (/api/*, etc.). The HTTP listener (80) is kept simple, with a single rule that just redirects to HTTPS.

Why it works

Each ALB costs about $16/month in fixed fees + LCU charges. Run 3 of them and that's $48/month in fixed fees alone. Even with traffic consolidated onto one ALB, LCU charges stay at roughly the same level, so the fixed cost simply drops to a third.

Another easily overlooked factor is the Public IPv4 charge introduced in February 2024. An ALB deployed across multiple AZs (3 AZs in our case) consumes one Public IPv4 address per AZ, so we were paying for 3 ALBs × 3 AZs = 9 Public IPv4 addresses (= $32.85/month). Consolidating into one brings that down to 3 addresses = $10.95/month, which means a hidden -$22/month saving on Public IPv4 charges buried in the VPC bucket happens at the same time.

In other words, the total impact of tactic #1 is "-$35/month on ELB" + "-$22/month on Public IPv4" ≈ about -$57/month. It's easy to miss if you only look at Cost Explorer's by-service view, so the existence of the Public IPv4 charge carries a lot of weight when deciding whether to consolidate ALBs.

Gotchas

• Listener rule priority design: with 23 rules lined up, unless you strictly follow the "more specific rules go higher" principle, traffic ends up at unintended backends. Grouping redirect rules at the end is the safe bet.
• Duplicate health check paths across target groups: if backends are designed to respond on the same path, health checks will still pass when traffic is mistakenly mapped to a different service, making it hard to notice. Use service-specific paths like /healthz.
• Carrying over WAF / access logs: if you forget to port the WAF and logging settings attached to the ALBs being merged over to the new ALB, you'll lose compliance coverage after consolidation.
• DNS cutover timing: shorten the Route 53 TTL in advance (to 60 seconds or so) and make sure you can roll back before switching the Alias record.

Before / After

3. Replacing NAT Gateway with an EC2 NAT (FCK-NAT)

What we did

We replaced the managed NAT Gateway with an EC2-based NAT instance (the FCK-NAT pattern). It's a single t3a.micro instance based on Amazon Linux 2023, wrapped in an Auto Scaling Group (min=max=desired=1) so that it recovers automatically.

The actual Launch Template / UserData / IAM policy / ASG settings we use in production, the real story of rewriting the Launch Template six times, and the operational caveats are all covered in a separate article, "Pinning the Outbound IP of ECS ── Our Record of Adopting fck-nat in Production". This article focuses on verifying the cost savings.

I said "Auto Scaling", but in practice it's self-healing, not scaling. The moment the EC2 health check goes unhealthy, the ASG launches a replacement instance from the same launch template. All you have to do is point 0.0.0.0/0 → the NAT instance's ENI (or the instance) in the private subnet's route table. The FCK-NAT AMI handles everything built in, from IP forwarding and the NAT iptables rules to disabling the ENI's SourceDestCheck, so there's no need to touch OS settings on the server by hand. For Public IPv4, we associate an EIP to get a fixed IP.

Why it works (verified against moobon's actual bill)

A managed NAT Gateway in ap-northeast-1 costs $0.045/h × 720h = about $32/month in hourly charges, plus $0.045/GB of data processed. That's all there is to it "on paper". But looking at moobon's actual bill, outbound traffic (ECR pulls / OS patches / ECS task launches / external API calls, etc.) was sending the equivalent of 1.4 TB every month through the NAT, and we were paying $62 a month in data processing alone.

We thought it was around "$32 fixed + a few tens of dollars in data processing", but in reality we were paying over $100 a month for a single NAT GW. While we vaguely used it because "managed means convenient", the data processing charges had been creeping up ── that's what was really going on.
A t3a.micro + 8GB EBS NAT EC2, on the other hand, comes to about $8/month in total for the instance plus storage. EC2 data transfer is free within an AZ / within a region, and only traffic leaving the AZ is charged, so the billing structure stays light even when traffic is heavy.

Gotchas and availability

• "Down" doesn't mean "every request fails": even if the NAT EC2 goes down, the only thing affected is traffic from servers to the outside world (API calls / OS package updates / ECR pulls / integrations with external SaaS). Requests that complete entirely within ALB → app → DB (in-VPC resources like RDS / ElastiCache) are unaffected. In other words, whether a NAT outage immediately breaks user-facing responses depends on how your system is built. If most of your web app is designed to complete within in-VPC resources, a NAT failure stays limited to "only the processes that depend on external services partially fail".
• Recovery time: ASG self-healing takes 2–4 minutes. That's worse than the managed NAT Gateway's 99.99% SLA, but as explained above, it's "not a total outage" ── "only features that need outbound traffic stop working temporarily". Depending on requirements, that's acceptable even in production in many cases.
• The only configuration is the route table: the FCK-NAT AMI takes care of IP forwarding / iptables / disabling SourceDestCheck, so you never have to touch OS settings on the server. All you do is write one line in the route table: 0.0.0.0/0 → the NAT ENI. It's much less error-prone than building a NAT server by hand from Amazon Linux.
• ENI switchover when the ASG replaces the instance: when the instance is replaced, the ENI changes. FCK-NAT ships with a startup script that automatically re-registers its own ENI in the route table. Check in advance that the route table ID and tags are passed correctly via the launch template.
• Bandwidth and connection limits: t3a.micro is burstable with a maximum of 5 Gbps, and its conntrack limit is modest too. For workloads that expect constant high traffic or large numbers of concurrent connections, consider t3.small or larger, or a network-optimized instance type.
• Monitoring: set CloudWatch alarms on the NAT instance's NetworkOut / CPUUtilization, as well as an application-side metric for "external API call success rate". Since it isn't managed, this prevents the "we suddenly noticed only the external integrations were broken" scenario.

Before / After

4. Consolidating multiple WAFs into 1

What we did (as a by-product of tactic #1)

This tactic was a by-product that wasn't in the original cost reduction plan. Once tactic #1 consolidated the ALBs from 3 to 1, it was only natural to consolidate the WAF Web ACLs attached to each ALB into one as well. In the process of merging the Web ACLs, we discovered duplicate rules and mismatched modes (a mix of Count / Block), so we cleaned up the rule configuration at the same time. The consolidated setup consists of these 8 rules:

Why it works

A WAF Web ACL is billed at a $5/month base fee per ACL, plus $1/month per rule and $0.60 per million requests. Run multiple WAFs and the base fee stacks up with each one. If the same managed rule group is loaded into multiple ACLs, you're also paying rule charges for each of them ── an inefficiency that consolidation eliminates.

Gotchas

• Sequence it with the ALB consolidation: doing it alongside the ALB consolidation in #1 makes the work easier, since there are fewer resources to attach to.
• Custom rule priority: evaluate the managed rule groups (P0–P5) first and put our own geo-filtering custom rules after them. In reverse order, you risk blocking legitimate requests by mistake.
• Check the cost of BotControl: AWSManagedRulesBotControlRuleSet in standard mode comes with an additional $10/month + request charges (priced differently from the other managed rules). Decide whether to turn it on or off based on your requirements.

Before / After

5. Moving ECS to private subnets (cutting Public IPv4)

What we did

We switched our ECS tasks from public subnets with directly assigned public IPs to private subnets with outbound traffic via NAT (the EC2 NAT). This eliminated the need for the Public IPv4 address that had been assigned to each task. Combined with the addresses cut by the ALB consolidation in #1, the number of billable Public IPv4 addresses dropped significantly.

Why it works

In February 2024, AWS started charging $0.005/h (≈ $3.65/month) for every Public IPv4 address. IPs that used to be handed out casually via "Auto-assign Public IP on the ENI" all became billable. In the month of writing (2026-04), MOOBON's APN1-PublicIPv4:InUseAddress came to $97.79 (the equivalent of 27 addresses per month), and APN1-PublicIPv4:IdleAddress to $5.51. Even with just 10 ECS tasks, a public placement costs $36 every month "just for holding IPs".

Gotchas

• ECR pulls go through the NAT: with private subnets, ECS tasks go through the NAT when pulling images from ECR. After switching to the EC2 NAT (tactic #2), the transfer cost itself is light, but Fargate task startup times tend to get slightly longer due to the NAT hop. If that matters at your scale, you could add Interface VPC endpoints for ecr.api / ecr.dkr / s3 / logs / secretsmanager to bypass the NAT, but we haven't evaluated that yet (it's listed under what we plan to tackle next).
• IP allowlists for external APIs (where this actually becomes the right architecture): payment APIs in particular (credit card, convenience store payment, bank transfer integrations, etc.) often require you to register and pin your source IP in advance as a business requirement. The thing to realize here is that the public IP assigned to an ECS Fargate task is dynamic and changes every time the task restarts, so it can't be used for allowlisting in the first place. With private subnets + outbound via the NAT's EIP, outbound traffic is funneled through the NAT's fixed EIP, which you can then register with the external party as your fixed IP. Alongside the cost savings, you automatically end up with the "right architecture" for external integrations that require allowlisting ── that's the other benefit of this tactic (for inbound APIs via the ALB, the ALB-side IPs apply as before).

Before / After

6. Trimming unnecessary CloudWatch Agent metrics

What we did

When we installed the CloudWatch Agent on our EC2 instances, we just ran it without really fine-tuning the configuration, so around 10 custom metrics per instance were being published. The only ones we actually used for alarms and incident analysis in practice were mem_used_percent and disk_used_percent, so we narrowed it down to just those two.

After the cleanup, moobon's config.json looks like this ── a minimal configuration with only mem and disk (retrieved from the actual instance via SSM to confirm):

{
  "agent": {"metrics_collection_interval": 60},
  "metrics": {
    "append_dimensions": {"InstanceId": "${aws:InstanceId}"},
    "metrics_collected": {
      "mem":  {"measurement": ["mem_used_percent"]},
      "disk": {"measurement": ["disk_used_percent"], "resources": ["/"]}
    }
  }
}

Why it works: from 10 per instance down to 2

When we started the cleanup, thanks to a haphazard initial setup, around 10 custom metrics per instance were being published. We cut this down to 2: mem + disk, used the standard EC2 metrics (free) for CPU, and stopped publishing disk IO, netstat, swap, processes, and the like, since we rarely used them.

In absolute terms it's a small -$19/month, but CloudWatch custom metrics cost $0.30 per metric per month (for the first 10K) and grow in direct proportion to the number of instances. The habit of "keeping collection to the bare minimum" pays off proportionally as your fleet grows, so it's worth enforcing from day one.

Gotchas: none

For this tactic, there really aren't any gotchas. There's no point collecting custom metrics you don't use for alarms, and just continuing to collect the data consumes resources (a resident CWAgent + network + metric charges). It's perfectly fine to delete unused metrics first thing. Even if you keep them around because "we might need them someday", you can start publishing them right away by configuring them when the time comes.

7. Migrating EC2 from AL2 to AL2023 + gp3 + smaller EBS

I've written a separate, detailed article on this one. The highlights: we shrank EBS from 100GB gp2 to 30GB gp3, offloaded unnecessary backups and logs to S3 to clear out everything that had piled up on EBS, and costed out keeping 10 generations of AMI snapshots. For moobon alone, EC2 - Other went from $166 to $41 (-$125 / -75%), making it one of the biggest cuts in this whole project.

Before / After (contribution to this project)

8. Moving staging ECS to Fargate Spot

What we did

We switched the ECS tasks in our staging environment, used for pre-release testing, to Fargate Spot. We either set FARGATE_SPOT to 100% in the capacity provider strategy, or used base 0 + weight 1, so that every task launches on Spot.

Why it works

Fargate Spot is up to 70% cheaper than regular Fargate. For a 0.25 vCPU / 0.5 GB task in ap-northeast-1, regular Fargate is $0.0123/hour → Spot is around $0.0037/hour (varies over time). Per month, that's roughly $9 → $2.7 per task. If you keep about 5 staging tasks running at all times, that works out to nearly $30 off every month.

Gotchas (no issues in our case)

• 2-minute interruption notice: Spot tasks can be interrupted at AWS's discretion, but ECS restarts them automatically. For staging, it's a total non-issue (for production, it depends on the use case).
• Securing capacity: if you request large amounts of Spot capacity for an extremely high-demand instance type, you may not get it. It's well suited to designs that can scale up from a minimum task count of 0.
• Cases where we wouldn't recommend it for production: long-running jobs (video encoding, etc.) and APIs that require an SLA of 99.9% or higher. Long-running jobs have to start over when interrupted, and SLA-bound services lose SLA for the duration of the interruption.

9. Moving Fargate to EC2 to use our Reserved Instances

What we did

Some production workloads that originally ran on EC2 had been migrated to ECS Fargate, and as a result we ended up in an unexpected situation where EC2 Reserved Instances (RIs) we had bought in the past were sitting unused. RIs are a fixed cost you've already committed to, so you keep paying for them whether you use them or not.

So we moved the tasks we had been running on Fargate (three small 0.25 vCPU tasks + one 1 vCPU task, four in total) onto the spare t3a.small RIs as ECS on EC2 (switching the production ECS capacity provider to the EC2 launch type). I only meant to put the RIs to full use, but it ended up delivering far bigger savings than I expected.

Why it works: Fargate bills vCPU and RAM separately

What I realized when I actually did it was that Fargate pricing is higher than I had assumed. Fargate charges for vCPU and RAM separately by the hour, and the on-demand rates in ap-northeast-1 are as follows:

In our case, we took 4 Fargate tasks in total (3 small tasks at 0.25 vCPU + 0.5 GB and 1 task at 1 vCPU + 2 GB) and split them across 2 t3a.small RIs as ECS on EC2. Here's the estimate:

The workload itself didn't change at all; simply moving it from Fargate to EC2 on 3-year RIs cut nearly 80% of the monthly cost. The biggest lesson from this was that "the extra 'convenience fee' we pay for Fargate being managed was much larger than I had imagined". If you have spare RIs, just moving your Fargate tasks onto them works better than you'd expect.

As a side benefit, the 1 vCPU + 2 GB task effectively got a spec bump by landing on a t3a.small (2 vCPU + 2 GiB). Fargate is a billing model that allocates exactly the vCPU / memory you request, while ECS tasks on EC2 can use up whatever the host has free ── a subtle difference that matters more than you'd think when reasoning about pricing.

Gotchas

• Designing the ECS capacity provider: to use EC2 as a capacity provider, you need to provision instances with an ASG and link it to ECS. It's not like Fargate, where you just declare the number of tasks.
• Not enough free EC2 capacity during rolling deploys: if you pack tasks onto EC2 sized exactly to their requested specs, there's no room for the new tasks launched during a deployment, and the deployment gets stuck. We worked around this by temporarily scaling out the ASG before deploying to stand up a "spare EC2 for deployment" in advance. The proper approach is a design that places multiple small tasks on a larger EC2 instance (leaving headroom on each host). "Tight packing" to match your RI inventory comes back to bite you in the form of more complex deployments.
• EC2 maintenance is back on your plate: OS patches, AMI updates, ECS Agent version management, and other operational work come back. You give up the main advantage of Fargate (being managed).

10. Lessons ─ What we learned by doing it

1. The February 2024 Public IPv4 charge quietly adds up. It's $3.65/month per IP. Scattered across ALBs / NATs / ECS tasks / EIPs, it can balloon into double-digit dollars before you notice. Make a habit of asking "does this really need a public IP?" at design time.
2. The "convenience fee" for managed services is bigger than you think. NAT Gateway and Fargate are certainly convenient, but in our case NAT Gateway → FCK-NAT freed up $99/month and Fargate → EC2 (RIs) $62/month. Decide workload by workload on your SLA requirements and your willingness to pay the "convenience fee". For internal and staging systems, dropping managed services doesn't hurt.
3. Clean up the leftovers of past decisions. RIs left over after migrating from EC2 to Fargate, ALBs split up per service, WAFs set up separately for each ALB ── "legacy from past migrations and setups" lingers as quiet fixed costs. Cleaning up often pays off more than new optimizations.
4. Working alongside AI lets you "learn while doing". From reading the bill to choosing how to cut costs, identifying unused services, and working around problems we hit, we consulted AI at every step of this project. Even for "costs I hadn't noticed" and "services I'd never touched", I could look into pricing, alternatives, and gotchas all at once, and as a result I was able to learn across the board while getting hands-on work done ── that was the most rewarding part of this project for me.

11. What we plan to tackle next

Here are the items we plan to continue working on as of the time of writing (2026-05). Feel free to use them as a checklist if you're looking to improve an AWS environment of a similar size.

Priority items planned for the near term