Aug 02 09:32:58.455715 ip-10-0-130-182 systemd[1]: Starting Kubernetes Kubelet... Aug 02 09:32:58.799795 ip-10-0-130-182 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 02 09:32:58.799795 ip-10-0-130-182 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 02 09:32:58.799795 ip-10-0-130-182 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 02 09:32:58.799795 ip-10-0-130-182 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 02 09:32:58.799795 ip-10-0-130-182 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 02 09:32:58.801485 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.801402 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 02 09:32:58.807257 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807230 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 02 09:32:58.807257 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807255 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 02 09:32:58.807257 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807262 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807266 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807269 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807272 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807275 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807278 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807281 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807283 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807286 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807289 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807291 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807294 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807296 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807299 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807302 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807304 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807307 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807310 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807312 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807315 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 02 09:32:58.807400 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807318 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807320 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807323 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807327 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807331 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807334 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807337 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807340 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807343 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807348 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807351 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807354 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807357 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807359 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807362 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807365 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807367 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807370 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807372 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807375 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 02 09:32:58.807885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807377 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807380 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807383 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807385 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807388 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807390 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807394 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807397 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807400 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807403 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807406 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807409 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807412 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807415 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807418 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807421 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807423 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807426 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807429 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 02 09:32:58.808361 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807431 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807434 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807436 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807439 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807443 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807446 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807449 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807452 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807454 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807457 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807473 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807477 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807481 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807484 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807487 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807490 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807493 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807495 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807498 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807500 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 02 09:32:58.808846 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807503 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807505 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807512 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807515 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807518 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807520 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807523 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807525 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807528 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807530 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807533 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807535 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807538 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807540 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807543 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807545 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807548 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.807551 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808220 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808229 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 02 09:32:58.809323 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808233 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808236 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808239 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808242 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808244 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808247 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808250 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808252 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808255 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808258 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808260 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808263 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808265 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808268 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808270 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808273 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808276 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808279 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808281 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 02 09:32:58.809851 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808284 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808286 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808289 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808292 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808294 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808297 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808299 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808302 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808305 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808307 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808310 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808314 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808316 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808319 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808322 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808324 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808327 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808329 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808332 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808334 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 02 09:32:58.810316 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808337 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808339 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808343 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808346 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808349 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808352 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808354 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808357 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808360 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808363 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808366 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808369 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808371 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808374 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808377 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808379 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808382 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808384 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808386 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808389 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 02 09:32:58.810845 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808392 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808394 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808397 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808400 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808404 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808407 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808410 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808413 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808416 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808419 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808422 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808424 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808427 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808430 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808432 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808435 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808438 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808440 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808443 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808445 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 02 09:32:58.811341 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808449 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808451 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808455 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808457 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808476 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808480 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808483 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808486 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808488 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808491 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808494 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808496 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808499 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808502 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808504 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808507 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808510 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808513 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808963 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808969 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 02 09:32:58.811853 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808972 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808975 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808979 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808981 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808984 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808987 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808990 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808993 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808996 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.808998 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809001 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809004 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809006 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809008 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809011 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809014 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809017 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809019 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809023 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809025 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 02 09:32:58.812331 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809028 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809031 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809033 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809036 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809039 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809043 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809047 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809050 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809053 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809055 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809058 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809060 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809063 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809065 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809067 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809070 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809072 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809075 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809077 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 02 09:32:58.812832 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809080 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809082 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809085 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809087 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809090 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809092 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809094 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809097 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809100 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809102 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809105 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809107 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809110 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809113 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809115 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809118 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809123 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809125 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809128 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809131 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 02 09:32:58.813291 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809133 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809135 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809138 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809140 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809143 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809145 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809148 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809151 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809153 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809157 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809160 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809163 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809166 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809169 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809171 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809174 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809176 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809179 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809181 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809184 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 02 09:32:58.813828 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809186 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809189 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809192 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809195 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809197 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809201 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809204 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809206 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809210 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809214 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809217 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809220 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809223 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809225 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809228 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809230 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809233 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809236 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809303 2564 flags.go:64] FLAG: --address="0.0.0.0" Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809310 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 02 09:32:58.814312 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809315 2564 flags.go:64] FLAG: --anonymous-auth="true" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809320 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809324 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809327 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809331 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809335 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809338 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809341 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809344 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809348 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809350 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809353 2564 flags.go:64] FLAG: --cgroup-root="" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809357 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809360 2564 flags.go:64] FLAG: --client-ca-file="" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809363 2564 flags.go:64] FLAG: --cloud-provider="external" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809366 2564 flags.go:64] FLAG: --cluster-dns="[]" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809370 2564 flags.go:64] FLAG: --cluster-domain="" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809373 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809377 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809380 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809383 2564 flags.go:64] FLAG: --container-log-max-files="5" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809387 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809390 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 02 09:32:58.814850 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809393 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809396 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809399 2564 flags.go:64] FLAG: --contention-profiling="false" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809401 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809404 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809407 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809410 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809414 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809417 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809420 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809422 2564 flags.go:64] FLAG: --enable-load-reader="false" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809425 2564 flags.go:64] FLAG: --enable-server="true" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809428 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809432 2564 flags.go:64] FLAG: --event-burst="100" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809435 2564 flags.go:64] FLAG: --event-qps="50" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809437 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809440 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809443 2564 flags.go:64] FLAG: --eviction-hard="" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809446 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809449 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809452 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809455 2564 flags.go:64] FLAG: --eviction-soft="" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809458 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809474 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809480 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 02 09:32:58.815412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809484 2564 flags.go:64] FLAG: --experimental-mounter-path="" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809487 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809490 2564 flags.go:64] FLAG: --fail-swap-on="true" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809492 2564 flags.go:64] FLAG: --feature-gates="" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809496 2564 flags.go:64] FLAG: --file-check-frequency="20s" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809499 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809502 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809505 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809508 2564 flags.go:64] FLAG: --healthz-port="10248" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809511 2564 flags.go:64] FLAG: --help="false" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809514 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-130-182.ec2.internal" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809517 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809520 2564 flags.go:64] FLAG: --http-check-frequency="20s" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809523 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809526 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809530 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809535 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809537 2564 flags.go:64] FLAG: --image-service-endpoint="" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809540 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809543 2564 flags.go:64] FLAG: --kube-api-burst="100" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809546 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809549 2564 flags.go:64] FLAG: --kube-api-qps="50" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809552 2564 flags.go:64] FLAG: --kube-reserved="" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809554 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 02 09:32:58.816487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809557 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809560 2564 flags.go:64] FLAG: --kubelet-cgroups="" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809563 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809566 2564 flags.go:64] FLAG: --lock-file="" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809568 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809571 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809574 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809579 2564 flags.go:64] FLAG: --log-json-split-stream="false" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809583 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809586 2564 flags.go:64] FLAG: --log-text-split-stream="false" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809589 2564 flags.go:64] FLAG: --logging-format="text" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809592 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809595 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809598 2564 flags.go:64] FLAG: --manifest-url="" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809601 2564 flags.go:64] FLAG: --manifest-url-header="" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809605 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809608 2564 flags.go:64] FLAG: --max-open-files="1000000" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809611 2564 flags.go:64] FLAG: --max-pods="110" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809614 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809617 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809620 2564 flags.go:64] FLAG: --memory-manager-policy="None" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809623 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809626 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809628 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809633 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 02 09:32:58.817566 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809639 2564 flags.go:64] FLAG: --node-status-max-images="50" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809643 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809646 2564 flags.go:64] FLAG: --oom-score-adj="-999" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809649 2564 flags.go:64] FLAG: --pod-cidr="" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809651 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809657 2564 flags.go:64] FLAG: --pod-manifest-path="" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809659 2564 flags.go:64] FLAG: --pod-max-pids="-1" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809662 2564 flags.go:64] FLAG: --pods-per-core="0" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809665 2564 flags.go:64] FLAG: --port="10250" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809668 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809671 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0baae1b8bceabb7f6" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809674 2564 flags.go:64] FLAG: --qos-reserved="" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809677 2564 flags.go:64] FLAG: --read-only-port="10255" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809680 2564 flags.go:64] FLAG: --register-node="true" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809683 2564 flags.go:64] FLAG: --register-with-taints="" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809686 2564 flags.go:64] FLAG: --registry-burst="10" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809689 2564 flags.go:64] FLAG: --registry-qps="5" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809692 2564 flags.go:64] FLAG: --reserved-cpus="" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809695 2564 flags.go:64] FLAG: --reserved-memory="" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809698 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809701 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809704 2564 flags.go:64] FLAG: --rotate-certificates="false" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809706 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809709 2564 flags.go:64] FLAG: --runonce="false" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809712 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 02 09:32:58.818644 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809715 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809717 2564 flags.go:64] FLAG: --seccomp-default="false" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809720 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809723 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809726 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809729 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809733 2564 flags.go:64] FLAG: --storage-driver-password="root" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809736 2564 flags.go:64] FLAG: --storage-driver-secure="false" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809739 2564 flags.go:64] FLAG: --storage-driver-table="stats" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809741 2564 flags.go:64] FLAG: --storage-driver-user="root" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809744 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809747 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809750 2564 flags.go:64] FLAG: --system-cgroups="" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809752 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809757 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809760 2564 flags.go:64] FLAG: --tls-cert-file="" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809763 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809767 2564 flags.go:64] FLAG: --tls-min-version="" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809770 2564 flags.go:64] FLAG: --tls-private-key-file="" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809773 2564 flags.go:64] FLAG: --topology-manager-policy="none" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809775 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809778 2564 flags.go:64] FLAG: --topology-manager-scope="container" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809781 2564 flags.go:64] FLAG: --v="2" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809786 2564 flags.go:64] FLAG: --version="false" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809790 2564 flags.go:64] FLAG: --vmodule="" Aug 02 09:32:58.819585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809794 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.809797 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809906 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809909 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809913 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809916 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809920 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809923 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809926 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809929 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809931 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809934 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809936 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809941 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809943 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809946 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809949 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809951 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809954 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809956 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 02 09:32:58.820587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809959 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809962 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809964 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809967 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809969 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809972 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809975 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809977 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809980 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809983 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809987 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809990 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809993 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809996 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.809998 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810001 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810003 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810006 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810008 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810011 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 02 09:32:58.821309 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810013 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810016 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810019 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810022 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810025 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810029 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810033 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810035 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810038 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810040 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810043 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810045 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810048 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810051 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810053 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810056 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810058 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810061 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810063 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 02 09:32:58.821944 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810066 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810068 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810071 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810074 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810078 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810082 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810085 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810088 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810090 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810093 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810095 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810098 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810100 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810103 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810106 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810108 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810111 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810113 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810117 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 02 09:32:58.822428 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810120 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810123 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810126 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810128 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810130 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810133 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810135 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810138 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810141 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810143 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810146 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810149 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810151 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810154 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810156 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810159 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810165 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810167 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810170 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810172 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 02 09:32:58.822952 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810176 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810179 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.810181 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.810752 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.817728 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.817746 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.817774 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.817783 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817866 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817874 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817882 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817886 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817891 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817896 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817900 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 02 09:32:58.823683 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817905 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817909 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817914 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817918 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817922 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817927 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817931 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817935 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817939 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817943 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817948 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817953 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817957 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817962 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817966 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817970 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817974 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817979 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817983 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 02 09:32:58.824103 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817987 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.817994 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818000 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818006 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818011 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818016 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818021 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818026 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818030 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818035 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818041 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818046 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818052 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818057 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818062 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818067 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818072 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818076 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818080 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 02 09:32:58.824608 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818085 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818089 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818093 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818098 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818102 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818106 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818110 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818114 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818118 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818123 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818127 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818131 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818139 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818145 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818150 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818154 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818159 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818164 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818168 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818172 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 02 09:32:58.825155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818176 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818180 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818184 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818189 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818193 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818197 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818201 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818205 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818209 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818214 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818218 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818222 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818226 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818231 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818235 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818239 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818243 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818247 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818251 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818255 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 02 09:32:58.825778 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818260 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818264 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818268 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818272 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818276 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818281 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818285 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818289 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818294 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818299 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818303 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818308 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818312 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818316 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 02 09:32:58.826599 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.818323 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818515 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818523 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818528 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818532 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818537 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818542 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818547 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818551 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818556 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818560 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818564 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818568 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818572 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818576 2564 feature_gate.go:328] unrecognized feature gate: Example2 Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818581 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818585 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818589 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818593 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818597 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 02 09:32:58.827208 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818601 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818605 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818612 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818618 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818624 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818630 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818638 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818644 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818649 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818653 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818657 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818661 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818664 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818668 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818672 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818676 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818681 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818685 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818689 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 02 09:32:58.827725 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818693 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818697 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818701 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818705 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818709 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818713 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818718 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818722 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818726 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818730 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818734 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818738 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818744 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818750 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818755 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818759 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818764 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818768 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818775 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 02 09:32:58.828197 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818779 2564 feature_gate.go:328] unrecognized feature gate: Example Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818784 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818788 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818792 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818796 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818800 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818805 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818809 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818813 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818817 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818822 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818826 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818830 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818834 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818838 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818842 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818846 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818850 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818854 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818858 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 02 09:32:58.828706 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818862 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818867 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818871 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818875 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818879 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818883 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818887 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818891 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818895 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818900 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818904 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818908 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818913 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818918 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818922 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818926 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818930 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818934 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818938 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818942 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 02 09:32:58.829177 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818946 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 02 09:32:58.829684 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:58.818950 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 02 09:32:58.829684 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.818958 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 02 09:32:58.829684 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.819092 2564 server.go:962] "Client rotation is on, will bootstrap in background" Aug 02 09:32:58.829684 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.821588 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 02 09:32:58.829684 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.822883 2564 server.go:1019] "Starting client certificate rotation" Aug 02 09:32:58.829684 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.822988 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 02 09:32:58.829684 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.823024 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 02 09:32:58.829684 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.826046 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 02 09:32:58.830386 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.830371 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 02 09:32:58.838960 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.838943 2564 log.go:25] "Validated CRI v1 runtime API" Aug 02 09:32:58.840722 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.840707 2564 log.go:25] "Validated CRI v1 image API" Aug 02 09:32:58.842165 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.842149 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 02 09:32:58.844327 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.844308 2564 fs.go:135] Filesystem UUIDs: map[5a3eb42e-94e8-4763-b097-f7ca2785eb30:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 90354ed7-5dbe-47c3-91a0-fc8d11e041a8:/dev/nvme0n1p4] Aug 02 09:32:58.844397 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.844326 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 02 09:32:58.849065 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.848956 2564 manager.go:216] Machine: {Timestamp:2026-08-02 09:32:58.847853256 +0000 UTC m=+0.288743703 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3041304 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29c385d8d37b69dac5f1d0a132d469 SystemUUID:ec29c385-d8d3-7b69-dac5-f1d0a132d469 BootID:0988c0ec-e23f-43e1-ac49-e468780e9ddd Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c4:31:6a:cc:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c4:31:6a:cc:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:42:1f:9a:13:b4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 02 09:32:58.849065 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.849057 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 02 09:32:58.849186 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.849125 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 02 09:32:58.850043 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.850028 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 02 09:32:58.851931 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.851907 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 02 09:32:58.852057 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.851935 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-182.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 02 09:32:58.852555 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.852545 2564 topology_manager.go:138] "Creating topology manager with none policy" Aug 02 09:32:58.852585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.852557 2564 container_manager_linux.go:309] "Creating device plugin manager" Aug 02 09:32:58.852585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.852564 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 02 09:32:58.852585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.852577 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 02 09:32:58.853201 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.853192 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 02 09:32:58.855875 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.855865 2564 state_mem.go:36] "Initialized new in-memory state store" Aug 02 09:32:58.855992 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.855982 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 02 09:32:58.856027 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.856013 2564 kubelet.go:486] "Attempting to sync node with API server" Aug 02 09:32:58.856027 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.856023 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 02 09:32:58.856084 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.856037 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 02 09:32:58.856084 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.856046 2564 kubelet.go:398] "Adding apiserver pod source" Aug 02 09:32:58.856084 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.856063 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 02 09:32:58.856379 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.856367 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 02 09:32:58.856423 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.856387 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 02 09:32:58.858456 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.858436 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 02 09:32:58.859777 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.859756 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 02 09:32:58.859885 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.859806 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 02 09:32:58.860880 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860867 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 02 09:32:58.860934 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860883 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 02 09:32:58.860934 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860889 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 02 09:32:58.860934 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860894 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 02 09:32:58.860934 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860903 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 02 09:32:58.860934 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860908 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 02 09:32:58.860934 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860916 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 02 09:32:58.860934 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860921 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 02 09:32:58.860934 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860927 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 02 09:32:58.860934 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860932 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 02 09:32:58.861160 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860944 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 02 09:32:58.861160 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.860950 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 02 09:32:58.861637 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.861627 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 02 09:32:58.861637 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.861636 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 02 09:32:58.864722 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.864707 2564 server.go:1268] "Started kubelet" Aug 02 09:32:58.864932 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.864914 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-182.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 02 09:32:58.865395 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.864843 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 02 09:32:58.865608 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.865579 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 02 09:32:58.865608 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.865589 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-182.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 02 09:32:58.865614 ip-10-0-130-182 systemd[1]: Started Kubernetes Kubelet. Aug 02 09:32:58.865873 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.865831 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 02 09:32:58.865928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.865886 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 02 09:32:58.868965 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.868934 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 02 09:32:58.869969 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.869933 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 02 09:32:58.870665 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.870645 2564 server.go:310] "Adding debug handlers to kubelet server" Aug 02 09:32:58.872351 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.871404 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-182.ec2.internal.18c7f33348b93c66 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-182.ec2.internal,UID:ip-10-0-130-182.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-182.ec2.internal,},FirstTimestamp:2026-08-02 09:32:58.864671846 +0000 UTC m=+0.305562293,LastTimestamp:2026-08-02 09:32:58.864671846 +0000 UTC m=+0.305562293,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-182.ec2.internal,}" Aug 02 09:32:58.872545 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.872525 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 02 09:32:58.872545 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.872539 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 02 09:32:58.872678 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.872637 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:58.872983 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.872961 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 02 09:32:58.872983 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.872982 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 02 09:32:58.873139 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.873076 2564 reconstruct.go:97] "Volume reconstruction finished" Aug 02 09:32:58.873139 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.873087 2564 reconciler.go:29] "Reconciler: start to sync state" Aug 02 09:32:58.873256 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.873241 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 02 09:32:58.873356 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.873338 2564 factory.go:55] Registering systemd factory Aug 02 09:32:58.873434 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.873357 2564 factory.go:223] Registration of the systemd container factory successfully Aug 02 09:32:58.873571 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.873556 2564 factory.go:153] Registering CRI-O factory Aug 02 09:32:58.873654 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.873573 2564 factory.go:223] Registration of the crio container factory successfully Aug 02 09:32:58.873654 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.873622 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 02 09:32:58.873654 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.873647 2564 factory.go:103] Registering Raw factory Aug 02 09:32:58.873788 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.873663 2564 manager.go:1196] Started watching for new ooms in manager Aug 02 09:32:58.873976 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.873954 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kp5wp" Aug 02 09:32:58.874060 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.874048 2564 manager.go:349] Starting recovery of all containers Aug 02 09:32:58.880882 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.880856 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kp5wp" Aug 02 09:32:58.883541 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.882156 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 02 09:32:58.883541 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.882236 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-182.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 02 09:32:58.884136 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.884118 2564 manager.go:354] Recovery completed Aug 02 09:32:58.888172 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.888161 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 02 09:32:58.889960 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.889945 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Aug 02 09:32:58.890017 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.889971 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Aug 02 09:32:58.890017 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.889981 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Aug 02 09:32:58.890507 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.890493 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 02 09:32:58.890507 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.890505 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 02 09:32:58.890636 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.890497 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 02 09:32:58.890636 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.890520 2564 state_mem.go:36] "Initialized new in-memory state store" Aug 02 09:32:58.891146 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.891132 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Aug 02 09:32:58.891220 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.891161 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Aug 02 09:32:58.891220 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.891177 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Aug 02 09:32:58.891410 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.891397 2564 policy_none.go:49] "None policy: Start" Aug 02 09:32:58.891478 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.891417 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 02 09:32:58.891478 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.891445 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 02 09:32:58.892325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.892313 2564 policy_none.go:47] "Start" Aug 02 09:32:58.892633 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.892557 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-182.ec2.internal.18c7f3334a3b1823 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-182.ec2.internal,UID:ip-10-0-130-182.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-182.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-182.ec2.internal,},FirstTimestamp:2026-08-02 09:32:58.889959459 +0000 UTC m=+0.330849905,LastTimestamp:2026-08-02 09:32:58.889959459 +0000 UTC m=+0.330849905,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-182.ec2.internal,}" Aug 02 09:32:58.938655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.938519 2564 manager.go:337] "Starting Device Plugin manager" Aug 02 09:32:58.938725 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.938680 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 02 09:32:58.938725 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.938694 2564 server.go:87] "Starting device plugin registration server" Aug 02 09:32:58.938895 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.938884 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 02 09:32:58.938927 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.938897 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 02 09:32:58.939014 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.938996 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 02 09:32:58.939093 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.939079 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 02 09:32:58.939093 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.939093 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 02 09:32:58.939581 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.939561 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 02 09:32:58.939646 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.939606 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:58.970782 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.970763 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 02 09:32:58.971908 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.971892 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 02 09:32:58.971966 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.971932 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 02 09:32:58.971966 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.971952 2564 kubelet.go:2448] "Starting kubelet main sync loop" Aug 02 09:32:58.972043 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:58.971991 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 02 09:32:58.974073 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:58.974058 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 02 09:32:59.039649 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.039632 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 02 09:32:59.040325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.040311 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Aug 02 09:32:59.040388 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.040337 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Aug 02 09:32:59.040388 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.040347 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Aug 02 09:32:59.040388 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.040370 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.048056 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.048038 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.048099 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.048058 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-182.ec2.internal\": node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:59.061501 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.061448 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:59.072547 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.072516 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"] Aug 02 09:32:59.072624 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.072588 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 02 09:32:59.073755 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.073741 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Aug 02 09:32:59.073814 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.073768 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Aug 02 09:32:59.073814 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.073777 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Aug 02 09:32:59.074798 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.074775 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 02 09:32:59.074902 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.074889 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.074939 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.074917 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 02 09:32:59.075657 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.075641 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Aug 02 09:32:59.075716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.075665 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Aug 02 09:32:59.075716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.075675 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Aug 02 09:32:59.075775 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.075649 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Aug 02 09:32:59.075775 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.075735 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Aug 02 09:32:59.075775 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.075748 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Aug 02 09:32:59.076652 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.076639 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.076697 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.076668 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 02 09:32:59.077351 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.077335 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Aug 02 09:32:59.077408 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.077365 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Aug 02 09:32:59.077408 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.077379 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Aug 02 09:32:59.103114 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.103095 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-182.ec2.internal\" not found" node="ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.107544 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.107527 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-182.ec2.internal\" not found" node="ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.162579 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.162552 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:59.174270 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.174251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eb38c9524ad6c33c50f346789dd6a0d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"eb38c9524ad6c33c50f346789dd6a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.174317 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.174276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb38c9524ad6c33c50f346789dd6a0d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"eb38c9524ad6c33c50f346789dd6a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.174317 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.174295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9cf72a7270824af7232dadef49144540-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"9cf72a7270824af7232dadef49144540\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.263177 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.263156 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:59.274766 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.274736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9cf72a7270824af7232dadef49144540-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"9cf72a7270824af7232dadef49144540\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.274845 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.274771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eb38c9524ad6c33c50f346789dd6a0d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"eb38c9524ad6c33c50f346789dd6a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.274845 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.274790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb38c9524ad6c33c50f346789dd6a0d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"eb38c9524ad6c33c50f346789dd6a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.274845 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.274828 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb38c9524ad6c33c50f346789dd6a0d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"eb38c9524ad6c33c50f346789dd6a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.274845 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.274840 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9cf72a7270824af7232dadef49144540-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"9cf72a7270824af7232dadef49144540\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.274983 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.274855 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eb38c9524ad6c33c50f346789dd6a0d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"eb38c9524ad6c33c50f346789dd6a0d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.363323 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.363281 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:59.406999 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.406970 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.409588 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.409573 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Aug 02 09:32:59.464245 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.464215 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:59.570472 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.570449 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:59.670607 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.670538 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:59.734804 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.734781 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 02 09:32:59.773336 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.773301 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:59.821921 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.821892 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 02 09:32:59.822330 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.822019 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 02 09:32:59.822330 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.822061 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 02 09:32:59.874272 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.874248 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:59.884849 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.884820 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-01 09:27:59 +0000 UTC" deadline="2028-02-27 11:34:00.864808183 +0000 UTC" Aug 02 09:32:59.884849 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.884846 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13778h1m0.979965794s" Aug 02 09:32:59.974349 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:32:59.974310 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:32:59.980937 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:59.980909 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb38c9524ad6c33c50f346789dd6a0d0.slice/crio-3e50cd8dc06eedf4a958d54c8a65e202054e3327b8455022f3fb486f6254089f WatchSource:0}: Error finding container 3e50cd8dc06eedf4a958d54c8a65e202054e3327b8455022f3fb486f6254089f: Status 404 returned error can't find the container with id 3e50cd8dc06eedf4a958d54c8a65e202054e3327b8455022f3fb486f6254089f Aug 02 09:32:59.990451 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:32:59.990436 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 02 09:32:59.996157 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:32:59.996134 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf72a7270824af7232dadef49144540.slice/crio-74cc4fb7ce1d34b8c45768c3ea66af57093ffb6a916098f1ed5d036bd0fce53c WatchSource:0}: Error finding container 74cc4fb7ce1d34b8c45768c3ea66af57093ffb6a916098f1ed5d036bd0fce53c: Status 404 returned error can't find the container with id 74cc4fb7ce1d34b8c45768c3ea66af57093ffb6a916098f1ed5d036bd0fce53c Aug 02 09:33:00.074574 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:00.074546 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:33:00.175717 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:00.175686 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Aug 02 09:33:00.241722 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.241658 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 02 09:33:00.251346 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.251320 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 02 09:33:00.273705 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.273679 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Aug 02 09:33:00.285145 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.285129 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 02 09:33:00.286174 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.286160 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Aug 02 09:33:00.292909 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.292894 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 02 09:33:00.857030 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.856997 2564 apiserver.go:52] "Watching apiserver" Aug 02 09:33:00.868174 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.868091 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 02 09:33:00.868601 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.868575 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-j7hkg","kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal","openshift-cluster-node-tuning-operator/tuned-g46pd","openshift-dns/node-resolver-dbgqp","openshift-multus/multus-additional-cni-plugins-4zlt5","openshift-multus/network-metrics-daemon-wr257","openshift-ovn-kubernetes/ovnkube-node-2vhs2","kube-system/global-pull-secret-syncer-qjr5v","kube-system/konnectivity-agent-n4td2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh","openshift-image-registry/node-ca-jxtkm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal","openshift-multus/multus-qv7kg","openshift-network-diagnostics/network-check-target-68nsw"] Aug 02 09:33:00.871402 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.871376 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.873426 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.873408 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 02 09:33:00.873659 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.873636 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 02 09:33:00.874204 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.873898 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 02 09:33:00.874204 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.874197 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.874461 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.874439 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 02 09:33:00.874606 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.874570 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 02 09:33:00.874722 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.874706 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 02 09:33:00.874799 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.874772 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 02 09:33:00.874925 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.874876 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-smt4n\"" Aug 02 09:33:00.876338 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.876316 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.877498 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.877467 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 02 09:33:00.877645 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.877498 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 02 09:33:00.877827 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.877805 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f6b46\"" Aug 02 09:33:00.878064 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.878046 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 02 09:33:00.878157 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.878070 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 02 09:33:00.878918 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.878714 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 02 09:33:00.878918 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.878722 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dbgqp" Aug 02 09:33:00.878918 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.878852 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j95mm\"" Aug 02 09:33:00.879128 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.878935 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 02 09:33:00.881407 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.880887 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 02 09:33:00.881407 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881278 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 02 09:33:00.881514 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881479 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sdzkv\"" Aug 02 09:33:00.881514 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881480 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-hostroot\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.881606 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvps\" (UniqueName: \"kubernetes.io/projected/c29d8355-7a1f-4575-86d5-4f027d749e11-kube-api-access-4rvps\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.881606 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881567 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-slash\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.881606 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-ovnkube-config\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.881712 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c29d8355-7a1f-4575-86d5-4f027d749e11-cni-binary-copy\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.881712 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-run-k8s-cni-cncf-io\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.881712 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-run-systemd\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.881712 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-etc-openvswitch\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.881864 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881712 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-run-openvswitch\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.881864 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.881864 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-env-overrides\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.881864 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881786 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-ovn-node-metrics-cert\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.881864 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rq5l\" (UniqueName: \"kubernetes.io/projected/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-kube-api-access-2rq5l\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.881864 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-kubelet\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.882058 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-system-cni-dir\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882058 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-var-lib-kubelet\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882058 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-conf-dir\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882058 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881944 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-etc-kubernetes\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882058 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-var-lib-openvswitch\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.882058 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-cni-bin\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.882058 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882033 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-ovnkube-script-lib\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.882367 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-os-release\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882367 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-var-lib-cni-multus\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882367 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-cni-dir\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882367 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882142 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-run-ovn\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.882367 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882168 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-log-socket\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.882367 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-daemon-config\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882367 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-run-multus-certs\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882367 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882268 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-systemd-units\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.882367 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.881765 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:00.882367 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-node-log\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.882367 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.882783 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882390 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-cni-netd\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.882783 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882414 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-cnibin\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882783 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882439 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-socket-dir-parent\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882783 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-run-netns\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882783 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-var-lib-cni-bin\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.882783 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.882583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-run-netns\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.884500 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.884443 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 02 09:33:00.884596 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.884525 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:00.884663 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:00.884617 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:00.884724 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.884688 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nv7dm\"" Aug 02 09:33:00.884783 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.884712 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 02 09:33:00.887046 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.887021 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 02 09:33:00.887817 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.887799 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:00.890282 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.890058 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qfb6w\"" Aug 02 09:33:00.890282 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.890152 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 02 09:33:00.890500 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.890328 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j7hkg" Aug 02 09:33:00.890500 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.890417 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 02 09:33:00.892730 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.892710 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 02 09:33:00.893021 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.892951 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 02 09:33:00.894599 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.893595 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 02 09:33:00.894599 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.893610 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n9qjp\"" Aug 02 09:33:00.894599 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.893915 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:00.894599 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:00.894100 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:00.898607 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.898587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:00.900754 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.900731 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 02 09:33:00.901115 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.901071 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 02 09:33:00.901418 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.901399 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jxtkm" Aug 02 09:33:00.901869 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.901850 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vztqm\"" Aug 02 09:33:00.902257 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.902233 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 02 09:33:00.903971 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.903951 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:00.904067 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:00.904040 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:00.904296 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.904276 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 02 09:33:00.904380 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.904297 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 02 09:33:00.904442 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.904384 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gnhr4\"" Aug 02 09:33:00.904442 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.904276 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 02 09:33:00.913645 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.913628 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kcwfm" Aug 02 09:33:00.919673 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.919642 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kcwfm" Aug 02 09:33:00.974739 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.974710 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 02 09:33:00.977073 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.976990 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" event={"ID":"9cf72a7270824af7232dadef49144540","Type":"ContainerStarted","Data":"74cc4fb7ce1d34b8c45768c3ea66af57093ffb6a916098f1ed5d036bd0fce53c"} Aug 02 09:33:00.978283 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.978260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"eb38c9524ad6c33c50f346789dd6a0d0","Type":"ContainerStarted","Data":"3e50cd8dc06eedf4a958d54c8a65e202054e3327b8455022f3fb486f6254089f"} Aug 02 09:33:00.983692 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983672 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46n6\" (UniqueName: \"kubernetes.io/projected/4117d772-71c5-4e71-b5d1-f19a215a9e1f-kube-api-access-l46n6\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:00.983799 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983715 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-var-lib-openvswitch\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.983799 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-cni-bin\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.983799 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-kubernetes\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.983799 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983793 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6be5c331-cc91-4b21-b720-fbf73f0da999-host\") pod \"node-ca-jxtkm\" (UID: \"6be5c331-cc91-4b21-b720-fbf73f0da999\") " pod="openshift-image-registry/node-ca-jxtkm" Aug 02 09:33:00.984010 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983817 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-cni-bin\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.984010 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-var-lib-cni-multus\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.984010 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983874 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-var-lib-cni-multus\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.984010 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983831 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-var-lib-openvswitch\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.984010 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-registration-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:00.984010 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-sys-fs\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:00.984010 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-log-socket\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.984325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.983998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-daemon-config\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.984325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-var-lib-kubelet\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.984325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984097 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-log-socket\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.984325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984126 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-host\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.984325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984152 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb52n\" (UniqueName: \"kubernetes.io/projected/b528680d-7978-4bba-ad55-8c68bb7bd545-kube-api-access-jb52n\") pod \"iptables-alerter-j7hkg\" (UID: \"b528680d-7978-4bba-ad55-8c68bb7bd545\") " pod="openshift-network-operator/iptables-alerter-j7hkg" Aug 02 09:33:00.984325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:00.984325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-systemd-units\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.984325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.984325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984275 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-cnibin\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.984325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-hostroot\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.984325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984320 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984275 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-systemd-units\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b528680d-7978-4bba-ad55-8c68bb7bd545-host-slash\") pod \"iptables-alerter-j7hkg\" (UID: \"b528680d-7978-4bba-ad55-8c68bb7bd545\") " pod="openshift-network-operator/iptables-alerter-j7hkg" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-hostroot\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-cnibin\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27a5cbb8-9293-4a54-9016-a822e9633ed2-dbus\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-ovnkube-config\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-run-k8s-cni-cncf-io\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lz29\" (UniqueName: \"kubernetes.io/projected/dc472391-a90e-4235-a348-e96b0cec389f-kube-api-access-2lz29\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984612 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3310e94b-d9e4-4f11-8c91-b54387f17a5e-hosts-file\") pod \"node-resolver-dbgqp\" (UID: \"3310e94b-d9e4-4f11-8c91-b54387f17a5e\") " pod="openshift-dns/node-resolver-dbgqp" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-run-k8s-cni-cncf-io\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-run-systemd\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-etc-openvswitch\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-run-systemd\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984705 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-etc-openvswitch\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-run-openvswitch\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984755 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-run-openvswitch\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984787 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rq5l\" (UniqueName: \"kubernetes.io/projected/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-kube-api-access-2rq5l\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984802 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-etc-kubernetes\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-kubelet\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984872 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-kubelet\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984897 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-etc-kubernetes\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-system-cni-dir\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.984977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-var-lib-kubelet\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-lib-modules\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-device-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-ovnkube-script-lib\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-system-cni-dir\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-var-lib-kubelet\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985101 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-daemon-config\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985107 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc472391-a90e-4235-a348-e96b0cec389f-tmp\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-system-cni-dir\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:00.985863 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985234 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0457b78e-44a9-4746-a24c-22b355904bb3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jmns\" (UniqueName: \"kubernetes.io/projected/6be5c331-cc91-4b21-b720-fbf73f0da999-kube-api-access-4jmns\") pod \"node-ca-jxtkm\" (UID: \"6be5c331-cc91-4b21-b720-fbf73f0da999\") " pod="openshift-image-registry/node-ca-jxtkm" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b528680d-7978-4bba-ad55-8c68bb7bd545-iptables-alerter-script\") pod \"iptables-alerter-j7hkg\" (UID: \"b528680d-7978-4bba-ad55-8c68bb7bd545\") " pod="openshift-network-operator/iptables-alerter-j7hkg" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.985428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-os-release\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986043 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-ovnkube-script-lib\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986242 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-sysctl-d\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986277 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-ovnkube-config\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986290 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-sys\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986360 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-os-release\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-socket-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986437 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-cni-dir\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986487 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-run-ovn\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-run-multus-certs\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986572 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-os-release\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-run-ovn\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-node-log\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.986727 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986666 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-cni-netd\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986666 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-run-multus-certs\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986709 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-node-log\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986735 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-cni-netd\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986787 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-socket-dir-parent\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-run-netns\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-var-lib-cni-bin\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.986963 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-cni-dir\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-modprobe-d\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987039 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-run-netns\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987116 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-host-var-lib-cni-bin\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987152 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d42b000-3950-4ca8-b754-e38a28f50662-konnectivity-ca\") pod \"konnectivity-agent-n4td2\" (UID: \"3d42b000-3950-4ca8-b754-e38a28f50662\") " pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-run-netns\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvps\" (UniqueName: \"kubernetes.io/projected/c29d8355-7a1f-4575-86d5-4f027d749e11-kube-api-access-4rvps\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-sysconfig\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-run\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.987463 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-run-netns\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987371 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-slash\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c29d8355-7a1f-4575-86d5-4f027d749e11-cni-binary-copy\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987601 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-sysctl-conf\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0457b78e-44a9-4746-a24c-22b355904bb3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6be5c331-cc91-4b21-b720-fbf73f0da999-serviceca\") pod \"node-ca-jxtkm\" (UID: \"6be5c331-cc91-4b21-b720-fbf73f0da999\") " pod="openshift-image-registry/node-ca-jxtkm" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.987745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27a5cbb8-9293-4a54-9016-a822e9633ed2-kubelet-config\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988020 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-host-slash\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d42b000-3950-4ca8-b754-e38a28f50662-agent-certs\") pod \"konnectivity-agent-n4td2\" (UID: \"3d42b000-3950-4ca8-b754-e38a28f50662\") " pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988089 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc472391-a90e-4235-a348-e96b0cec389f-etc-tuned\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-env-overrides\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.988313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-ovn-node-metrics-cert\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.988980 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-conf-dir\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.988980 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-systemd\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:00.988980 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0457b78e-44a9-4746-a24c-22b355904bb3-cni-binary-copy\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:00.988980 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988781 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fpvl\" (UniqueName: \"kubernetes.io/projected/0457b78e-44a9-4746-a24c-22b355904bb3-kube-api-access-5fpvl\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:00.988980 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-cnibin\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:00.988980 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988903 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm52g\" (UniqueName: \"kubernetes.io/projected/9f8dcf84-140a-4955-933b-6f40794f9b9e-kube-api-access-jm52g\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:00.988980 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.988939 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-env-overrides\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.989324 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.989213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3310e94b-d9e4-4f11-8c91-b54387f17a5e-tmp-dir\") pod \"node-resolver-dbgqp\" (UID: \"3310e94b-d9e4-4f11-8c91-b54387f17a5e\") " pod="openshift-dns/node-resolver-dbgqp" Aug 02 09:33:00.989324 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.989244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxsgd\" (UniqueName: \"kubernetes.io/projected/3310e94b-d9e4-4f11-8c91-b54387f17a5e-kube-api-access-qxsgd\") pod \"node-resolver-dbgqp\" (UID: \"3310e94b-d9e4-4f11-8c91-b54387f17a5e\") " pod="openshift-dns/node-resolver-dbgqp" Aug 02 09:33:00.989324 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.989282 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvwv4\" (UniqueName: \"kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4\") pod \"network-check-target-68nsw\" (UID: \"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30\") " pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:00.989467 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.989408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-conf-dir\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.989834 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.989786 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 02 09:33:00.991402 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.991362 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c29d8355-7a1f-4575-86d5-4f027d749e11-cni-binary-copy\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.993032 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.992999 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c29d8355-7a1f-4575-86d5-4f027d749e11-multus-socket-dir-parent\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:00.994194 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.994165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-ovn-node-metrics-cert\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.995605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.995580 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rq5l\" (UniqueName: \"kubernetes.io/projected/a1bb0df9-1cc4-449a-8d50-76efb8b76a43-kube-api-access-2rq5l\") pod \"ovnkube-node-2vhs2\" (UID: \"a1bb0df9-1cc4-449a-8d50-76efb8b76a43\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:00.996431 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:00.996411 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvps\" (UniqueName: \"kubernetes.io/projected/c29d8355-7a1f-4575-86d5-4f027d749e11-kube-api-access-4rvps\") pod \"multus-qv7kg\" (UID: \"c29d8355-7a1f-4575-86d5-4f027d749e11\") " pod="openshift-multus/multus-qv7kg" Aug 02 09:33:01.090189 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090152 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27a5cbb8-9293-4a54-9016-a822e9633ed2-kubelet-config\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:01.090189 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d42b000-3950-4ca8-b754-e38a28f50662-agent-certs\") pod \"konnectivity-agent-n4td2\" (UID: \"3d42b000-3950-4ca8-b754-e38a28f50662\") " pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:01.090417 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc472391-a90e-4235-a348-e96b0cec389f-etc-tuned\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.090417 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-systemd\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.090417 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0457b78e-44a9-4746-a24c-22b355904bb3-cni-binary-copy\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.090417 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090272 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fpvl\" (UniqueName: \"kubernetes.io/projected/0457b78e-44a9-4746-a24c-22b355904bb3-kube-api-access-5fpvl\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.090417 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-cnibin\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.090417 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm52g\" (UniqueName: \"kubernetes.io/projected/9f8dcf84-140a-4955-933b-6f40794f9b9e-kube-api-access-jm52g\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:01.090417 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090331 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-systemd\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.090417 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3310e94b-d9e4-4f11-8c91-b54387f17a5e-tmp-dir\") pod \"node-resolver-dbgqp\" (UID: \"3310e94b-d9e4-4f11-8c91-b54387f17a5e\") " pod="openshift-dns/node-resolver-dbgqp" Aug 02 09:33:01.090417 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxsgd\" (UniqueName: \"kubernetes.io/projected/3310e94b-d9e4-4f11-8c91-b54387f17a5e-kube-api-access-qxsgd\") pod \"node-resolver-dbgqp\" (UID: \"3310e94b-d9e4-4f11-8c91-b54387f17a5e\") " pod="openshift-dns/node-resolver-dbgqp" Aug 02 09:33:01.090417 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwv4\" (UniqueName: \"kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4\") pod \"network-check-target-68nsw\" (UID: \"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30\") " pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46n6\" (UniqueName: \"kubernetes.io/projected/4117d772-71c5-4e71-b5d1-f19a215a9e1f-kube-api-access-l46n6\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-kubernetes\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6be5c331-cc91-4b21-b720-fbf73f0da999-host\") pod \"node-ca-jxtkm\" (UID: \"6be5c331-cc91-4b21-b720-fbf73f0da999\") " pod="openshift-image-registry/node-ca-jxtkm" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090524 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-registration-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-sys-fs\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090556 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-cnibin\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-var-lib-kubelet\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-host\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb52n\" (UniqueName: \"kubernetes.io/projected/b528680d-7978-4bba-ad55-8c68bb7bd545-kube-api-access-jb52n\") pod \"iptables-alerter-j7hkg\" (UID: \"b528680d-7978-4bba-ad55-8c68bb7bd545\") " pod="openshift-network-operator/iptables-alerter-j7hkg" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b528680d-7978-4bba-ad55-8c68bb7bd545-host-slash\") pod \"iptables-alerter-j7hkg\" (UID: \"b528680d-7978-4bba-ad55-8c68bb7bd545\") " pod="openshift-network-operator/iptables-alerter-j7hkg" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27a5cbb8-9293-4a54-9016-a822e9633ed2-dbus\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090747 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6be5c331-cc91-4b21-b720-fbf73f0da999-host\") pod \"node-ca-jxtkm\" (UID: \"6be5c331-cc91-4b21-b720-fbf73f0da999\") " pod="openshift-image-registry/node-ca-jxtkm" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-sys-fs\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090778 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lz29\" (UniqueName: \"kubernetes.io/projected/dc472391-a90e-4235-a348-e96b0cec389f-kube-api-access-2lz29\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090805 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-kubernetes\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.090928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3310e94b-d9e4-4f11-8c91-b54387f17a5e-hosts-file\") pod \"node-resolver-dbgqp\" (UID: \"3310e94b-d9e4-4f11-8c91-b54387f17a5e\") " pod="openshift-dns/node-resolver-dbgqp" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090863 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-registration-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-lib-modules\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090904 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-device-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090921 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0457b78e-44a9-4746-a24c-22b355904bb3-cni-binary-copy\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090932 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc472391-a90e-4235-a348-e96b0cec389f-tmp\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.090995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-system-cni-dir\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0457b78e-44a9-4746-a24c-22b355904bb3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jmns\" (UniqueName: \"kubernetes.io/projected/6be5c331-cc91-4b21-b720-fbf73f0da999-kube-api-access-4jmns\") pod \"node-ca-jxtkm\" (UID: \"6be5c331-cc91-4b21-b720-fbf73f0da999\") " pod="openshift-image-registry/node-ca-jxtkm" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3310e94b-d9e4-4f11-8c91-b54387f17a5e-tmp-dir\") pod \"node-resolver-dbgqp\" (UID: \"3310e94b-d9e4-4f11-8c91-b54387f17a5e\") " pod="openshift-dns/node-resolver-dbgqp" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b528680d-7978-4bba-ad55-8c68bb7bd545-iptables-alerter-script\") pod \"iptables-alerter-j7hkg\" (UID: \"b528680d-7978-4bba-ad55-8c68bb7bd545\") " pod="openshift-network-operator/iptables-alerter-j7hkg" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-sysctl-d\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-sys\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091135 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091168 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-var-lib-kubelet\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-socket-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.091815 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091218 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b528680d-7978-4bba-ad55-8c68bb7bd545-host-slash\") pod \"iptables-alerter-j7hkg\" (UID: \"b528680d-7978-4bba-ad55-8c68bb7bd545\") " pod="openshift-network-operator/iptables-alerter-j7hkg" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-os-release\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.091281 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091296 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27a5cbb8-9293-4a54-9016-a822e9633ed2-dbus\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-modprobe-d\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.091381 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs podName:9f8dcf84-140a-4955-933b-6f40794f9b9e nodeName:}" failed. No retries permitted until 2026-08-02 09:33:01.591342726 +0000 UTC m=+3.032233181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs") pod "network-metrics-daemon-wr257" (UID: "9f8dcf84-140a-4955-933b-6f40794f9b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d42b000-3950-4ca8-b754-e38a28f50662-konnectivity-ca\") pod \"konnectivity-agent-n4td2\" (UID: \"3d42b000-3950-4ca8-b754-e38a28f50662\") " pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.091459 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091511 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-sysconfig\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091458 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-os-release\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091529 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-system-cni-dir\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091518 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-host\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.091632 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret podName:27a5cbb8-9293-4a54-9016-a822e9633ed2 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:01.591614178 +0000 UTC m=+3.032504615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret") pod "global-pull-secret-syncer-qjr5v" (UID: "27a5cbb8-9293-4a54-9016-a822e9633ed2") : object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-sysconfig\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091665 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-run\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091671 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-socket-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.092716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-sys\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-sysctl-conf\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0457b78e-44a9-4746-a24c-22b355904bb3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6be5c331-cc91-4b21-b720-fbf73f0da999-serviceca\") pod \"node-ca-jxtkm\" (UID: \"6be5c331-cc91-4b21-b720-fbf73f0da999\") " pod="openshift-image-registry/node-ca-jxtkm" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.091896 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-sysctl-d\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092097 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0457b78e-44a9-4746-a24c-22b355904bb3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b528680d-7978-4bba-ad55-8c68bb7bd545-iptables-alerter-script\") pod \"iptables-alerter-j7hkg\" (UID: \"b528680d-7978-4bba-ad55-8c68bb7bd545\") " pod="openshift-network-operator/iptables-alerter-j7hkg" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-modprobe-d\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-run\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6be5c331-cc91-4b21-b720-fbf73f0da999-serviceca\") pod \"node-ca-jxtkm\" (UID: \"6be5c331-cc91-4b21-b720-fbf73f0da999\") " pod="openshift-image-registry/node-ca-jxtkm" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-etc-sysctl-conf\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092352 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3310e94b-d9e4-4f11-8c91-b54387f17a5e-hosts-file\") pod \"node-resolver-dbgqp\" (UID: \"3310e94b-d9e4-4f11-8c91-b54387f17a5e\") " pod="openshift-dns/node-resolver-dbgqp" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc472391-a90e-4235-a348-e96b0cec389f-lib-modules\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4117d772-71c5-4e71-b5d1-f19a215a9e1f-device-dir\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.093580 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27a5cbb8-9293-4a54-9016-a822e9633ed2-kubelet-config\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:01.094129 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0457b78e-44a9-4746-a24c-22b355904bb3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.094129 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d42b000-3950-4ca8-b754-e38a28f50662-konnectivity-ca\") pod \"konnectivity-agent-n4td2\" (UID: \"3d42b000-3950-4ca8-b754-e38a28f50662\") " pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:01.094129 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.092813 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0457b78e-44a9-4746-a24c-22b355904bb3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.094129 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.093367 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d42b000-3950-4ca8-b754-e38a28f50662-agent-certs\") pod \"konnectivity-agent-n4td2\" (UID: \"3d42b000-3950-4ca8-b754-e38a28f50662\") " pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:01.094129 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.093522 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc472391-a90e-4235-a348-e96b0cec389f-etc-tuned\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.094129 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.093653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc472391-a90e-4235-a348-e96b0cec389f-tmp\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.097067 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.097046 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 02 09:33:01.097211 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.097072 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 02 09:33:01.097211 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.097088 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bvwv4 for pod openshift-network-diagnostics/network-check-target-68nsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:01.097211 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.097150 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4 podName:75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:01.597135863 +0000 UTC m=+3.038026303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bvwv4" (UniqueName: "kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4") pod "network-check-target-68nsw" (UID: "75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:01.100392 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.099876 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm52g\" (UniqueName: \"kubernetes.io/projected/9f8dcf84-140a-4955-933b-6f40794f9b9e-kube-api-access-jm52g\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:01.100392 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.100341 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fpvl\" (UniqueName: \"kubernetes.io/projected/0457b78e-44a9-4746-a24c-22b355904bb3-kube-api-access-5fpvl\") pod \"multus-additional-cni-plugins-4zlt5\" (UID: \"0457b78e-44a9-4746-a24c-22b355904bb3\") " pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.100392 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.100355 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb52n\" (UniqueName: \"kubernetes.io/projected/b528680d-7978-4bba-ad55-8c68bb7bd545-kube-api-access-jb52n\") pod \"iptables-alerter-j7hkg\" (UID: \"b528680d-7978-4bba-ad55-8c68bb7bd545\") " pod="openshift-network-operator/iptables-alerter-j7hkg" Aug 02 09:33:01.101156 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.101125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxsgd\" (UniqueName: \"kubernetes.io/projected/3310e94b-d9e4-4f11-8c91-b54387f17a5e-kube-api-access-qxsgd\") pod \"node-resolver-dbgqp\" (UID: \"3310e94b-d9e4-4f11-8c91-b54387f17a5e\") " pod="openshift-dns/node-resolver-dbgqp" Aug 02 09:33:01.101343 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.101324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jmns\" (UniqueName: \"kubernetes.io/projected/6be5c331-cc91-4b21-b720-fbf73f0da999-kube-api-access-4jmns\") pod \"node-ca-jxtkm\" (UID: \"6be5c331-cc91-4b21-b720-fbf73f0da999\") " pod="openshift-image-registry/node-ca-jxtkm" Aug 02 09:33:01.101979 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.101956 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lz29\" (UniqueName: \"kubernetes.io/projected/dc472391-a90e-4235-a348-e96b0cec389f-kube-api-access-2lz29\") pod \"tuned-g46pd\" (UID: \"dc472391-a90e-4235-a348-e96b0cec389f\") " pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.102074 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.102058 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46n6\" (UniqueName: \"kubernetes.io/projected/4117d772-71c5-4e71-b5d1-f19a215a9e1f-kube-api-access-l46n6\") pod \"aws-ebs-csi-driver-node-b9ckh\" (UID: \"4117d772-71c5-4e71-b5d1-f19a215a9e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.186420 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.186346 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:01.195642 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.195620 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qv7kg" Aug 02 09:33:01.205689 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.205656 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g46pd" Aug 02 09:33:01.210505 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.210482 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dbgqp" Aug 02 09:33:01.218253 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.218234 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" Aug 02 09:33:01.227108 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.227088 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:01.235747 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.235724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j7hkg" Aug 02 09:33:01.242882 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.242865 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" Aug 02 09:33:01.249703 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.249689 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jxtkm" Aug 02 09:33:01.384991 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.384962 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 02 09:33:01.467497 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.467392 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 02 09:33:01.597218 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.597175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:01.597390 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.597230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:01.597390 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.597344 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:01.597390 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.597353 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:01.597540 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.597415 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret podName:27a5cbb8-9293-4a54-9016-a822e9633ed2 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:02.597394862 +0000 UTC m=+4.038285324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret") pod "global-pull-secret-syncer-qjr5v" (UID: "27a5cbb8-9293-4a54-9016-a822e9633ed2") : object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:01.597540 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.597431 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs podName:9f8dcf84-140a-4955-933b-6f40794f9b9e nodeName:}" failed. No retries permitted until 2026-08-02 09:33:02.597424277 +0000 UTC m=+4.038314713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs") pod "network-metrics-daemon-wr257" (UID: "9f8dcf84-140a-4955-933b-6f40794f9b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:01.698540 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.698507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwv4\" (UniqueName: \"kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4\") pod \"network-check-target-68nsw\" (UID: \"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30\") " pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:01.698745 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.698654 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 02 09:33:01.698745 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.698673 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 02 09:33:01.698745 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.698684 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bvwv4 for pod openshift-network-diagnostics/network-check-target-68nsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:01.698745 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:01.698741 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4 podName:75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:02.698721074 +0000 UTC m=+4.139611510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvwv4" (UniqueName: "kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4") pod "network-check-target-68nsw" (UID: "75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:01.923741 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.923665 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-01 09:28:01 +0000 UTC" deadline="2028-03-01 15:15:49.341674541 +0000 UTC" Aug 02 09:33:01.923741 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.923695 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13853h42m47.417983998s" Aug 02 09:33:01.959279 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:33:01.959255 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3310e94b_d9e4_4f11_8c91_b54387f17a5e.slice/crio-964dffb091545bd6d7b7402800f281943ac6f191b5a0e324a7b4a3befb147b80 WatchSource:0}: Error finding container 964dffb091545bd6d7b7402800f281943ac6f191b5a0e324a7b4a3befb147b80: Status 404 returned error can't find the container with id 964dffb091545bd6d7b7402800f281943ac6f191b5a0e324a7b4a3befb147b80 Aug 02 09:33:01.960928 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:33:01.960856 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0457b78e_44a9_4746_a24c_22b355904bb3.slice/crio-c7b1dd531e7be0f5c9d1fdaad929b767332e12edd0d8f2c28793613de365c43c WatchSource:0}: Error finding container c7b1dd531e7be0f5c9d1fdaad929b767332e12edd0d8f2c28793613de365c43c: Status 404 returned error can't find the container with id c7b1dd531e7be0f5c9d1fdaad929b767332e12edd0d8f2c28793613de365c43c Aug 02 09:33:01.965378 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:33:01.965355 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc472391_a90e_4235_a348_e96b0cec389f.slice/crio-9689990056a903734d2b4ee67042d1290f7a38d205c052d1d5cbbba0f80d986a WatchSource:0}: Error finding container 9689990056a903734d2b4ee67042d1290f7a38d205c052d1d5cbbba0f80d986a: Status 404 returned error can't find the container with id 9689990056a903734d2b4ee67042d1290f7a38d205c052d1d5cbbba0f80d986a Aug 02 09:33:01.967495 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:33:01.967417 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29d8355_7a1f_4575_86d5_4f027d749e11.slice/crio-059878f03e2fb56e7ba1837a964d695add13fc52d1b5ae4f2337a5a2065a1585 WatchSource:0}: Error finding container 059878f03e2fb56e7ba1837a964d695add13fc52d1b5ae4f2337a5a2065a1585: Status 404 returned error can't find the container with id 059878f03e2fb56e7ba1837a964d695add13fc52d1b5ae4f2337a5a2065a1585 Aug 02 09:33:01.969254 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:33:01.969227 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb528680d_7978_4bba_ad55_8c68bb7bd545.slice/crio-4e4e1783ed20203b779ccc52c139c75b772d4731cd214fc2e2227e121fa558e1 WatchSource:0}: Error finding container 4e4e1783ed20203b779ccc52c139c75b772d4731cd214fc2e2227e121fa558e1: Status 404 returned error can't find the container with id 4e4e1783ed20203b779ccc52c139c75b772d4731cd214fc2e2227e121fa558e1 Aug 02 09:33:01.981468 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.981440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" event={"ID":"a1bb0df9-1cc4-449a-8d50-76efb8b76a43","Type":"ContainerStarted","Data":"02e0c1259396b45b36db685d7882814a6cf157bdc7943ab854d98d8751a1dcbb"} Aug 02 09:33:01.983224 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.983147 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" event={"ID":"4117d772-71c5-4e71-b5d1-f19a215a9e1f","Type":"ContainerStarted","Data":"9dec0d43a35a88f8f83e41a8f8cae83835f91c1bbe46e3cec6b0a35384071f73"} Aug 02 09:33:01.984750 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.984726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n4td2" event={"ID":"3d42b000-3950-4ca8-b754-e38a28f50662","Type":"ContainerStarted","Data":"88b39746b5289332bb00e6e1af2cb1b37901b4789be402863eca11b176d25b22"} Aug 02 09:33:01.986205 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.986179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jxtkm" event={"ID":"6be5c331-cc91-4b21-b720-fbf73f0da999","Type":"ContainerStarted","Data":"19caedfe901c1b5737572f89d65a6bebe87621a1a44c9cb30919249bb89bcab1"} Aug 02 09:33:01.987587 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.987566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dbgqp" event={"ID":"3310e94b-d9e4-4f11-8c91-b54387f17a5e","Type":"ContainerStarted","Data":"964dffb091545bd6d7b7402800f281943ac6f191b5a0e324a7b4a3befb147b80"} Aug 02 09:33:01.988780 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.988757 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j7hkg" event={"ID":"b528680d-7978-4bba-ad55-8c68bb7bd545","Type":"ContainerStarted","Data":"4e4e1783ed20203b779ccc52c139c75b772d4731cd214fc2e2227e121fa558e1"} Aug 02 09:33:01.994453 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.994429 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qv7kg" event={"ID":"c29d8355-7a1f-4575-86d5-4f027d749e11","Type":"ContainerStarted","Data":"059878f03e2fb56e7ba1837a964d695add13fc52d1b5ae4f2337a5a2065a1585"} Aug 02 09:33:01.995601 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.995577 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g46pd" event={"ID":"dc472391-a90e-4235-a348-e96b0cec389f","Type":"ContainerStarted","Data":"9689990056a903734d2b4ee67042d1290f7a38d205c052d1d5cbbba0f80d986a"} Aug 02 09:33:01.996678 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:01.996660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" event={"ID":"0457b78e-44a9-4746-a24c-22b355904bb3","Type":"ContainerStarted","Data":"c7b1dd531e7be0f5c9d1fdaad929b767332e12edd0d8f2c28793613de365c43c"} Aug 02 09:33:02.606271 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:02.605768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:02.606271 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:02.606038 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:02.606271 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:02.606085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:02.606271 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:02.606106 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs podName:9f8dcf84-140a-4955-933b-6f40794f9b9e nodeName:}" failed. No retries permitted until 2026-08-02 09:33:04.606085785 +0000 UTC m=+6.046976228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs") pod "network-metrics-daemon-wr257" (UID: "9f8dcf84-140a-4955-933b-6f40794f9b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:02.606271 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:02.606154 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:02.606271 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:02.606192 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret podName:27a5cbb8-9293-4a54-9016-a822e9633ed2 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:04.606181056 +0000 UTC m=+6.047071493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret") pod "global-pull-secret-syncer-qjr5v" (UID: "27a5cbb8-9293-4a54-9016-a822e9633ed2") : object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:02.707410 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:02.707373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwv4\" (UniqueName: \"kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4\") pod \"network-check-target-68nsw\" (UID: \"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30\") " pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:02.707619 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:02.707577 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 02 09:33:02.707619 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:02.707597 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 02 09:33:02.707619 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:02.707611 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bvwv4 for pod openshift-network-diagnostics/network-check-target-68nsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:02.707792 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:02.707668 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4 podName:75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:04.707649857 +0000 UTC m=+6.148540296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvwv4" (UniqueName: "kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4") pod "network-check-target-68nsw" (UID: "75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:02.924661 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:02.924535 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-01 09:28:01 +0000 UTC" deadline="2028-03-08 17:46:50.443540045 +0000 UTC" Aug 02 09:33:02.924661 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:02.924601 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14024h13m47.518943457s" Aug 02 09:33:02.977174 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:02.977143 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:02.977338 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:02.977291 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:02.977765 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:02.977741 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:02.977881 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:02.977847 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:02.977954 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:02.977939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:02.978042 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:02.978022 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:03.009590 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:03.009560 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" event={"ID":"9cf72a7270824af7232dadef49144540","Type":"ContainerStarted","Data":"e2a823df41f951ad38ab937b7220a6e2ba2858441e2754f26616b2f33ad9d0d7"} Aug 02 09:33:03.020103 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:03.020075 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb38c9524ad6c33c50f346789dd6a0d0" containerID="8e61ec9966989bc99eadd736aa650d10cbadffeadbd741aa71d3f1b317c4171c" exitCode=0 Aug 02 09:33:03.020212 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:03.020125 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"eb38c9524ad6c33c50f346789dd6a0d0","Type":"ContainerDied","Data":"8e61ec9966989bc99eadd736aa650d10cbadffeadbd741aa71d3f1b317c4171c"} Aug 02 09:33:03.046438 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:03.046051 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" podStartSLOduration=3.046031019 podStartE2EDuration="3.046031019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:03.023955357 +0000 UTC m=+4.464845818" watchObservedRunningTime="2026-08-02 09:33:03.046031019 +0000 UTC m=+4.486921480" Aug 02 09:33:04.041761 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:04.041634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"eb38c9524ad6c33c50f346789dd6a0d0","Type":"ContainerStarted","Data":"1fe3ad6f65619c7c7e165dde827d1936fb4037d9288fc2c33722a58394702705"} Aug 02 09:33:04.625483 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:04.625433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:04.625681 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:04.625492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:04.625744 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:04.625703 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:04.625744 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:04.625710 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:04.625866 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:04.625784 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret podName:27a5cbb8-9293-4a54-9016-a822e9633ed2 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:08.62576267 +0000 UTC m=+10.066653121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret") pod "global-pull-secret-syncer-qjr5v" (UID: "27a5cbb8-9293-4a54-9016-a822e9633ed2") : object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:04.625866 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:04.625807 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs podName:9f8dcf84-140a-4955-933b-6f40794f9b9e nodeName:}" failed. No retries permitted until 2026-08-02 09:33:08.625795073 +0000 UTC m=+10.066685509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs") pod "network-metrics-daemon-wr257" (UID: "9f8dcf84-140a-4955-933b-6f40794f9b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:04.726548 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:04.726157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwv4\" (UniqueName: \"kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4\") pod \"network-check-target-68nsw\" (UID: \"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30\") " pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:04.726548 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:04.726349 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 02 09:33:04.726548 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:04.726370 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 02 09:33:04.726548 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:04.726383 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bvwv4 for pod openshift-network-diagnostics/network-check-target-68nsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:04.726548 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:04.726440 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4 podName:75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:08.726421656 +0000 UTC m=+10.167312096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvwv4" (UniqueName: "kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4") pod "network-check-target-68nsw" (UID: "75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:04.973719 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:04.973637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:04.973900 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:04.973793 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:04.973900 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:04.973637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:04.973900 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:04.973873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:04.974074 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:04.973993 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:04.974129 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:04.974079 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:06.975377 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:06.975310 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:06.975849 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:06.975448 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:06.975928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:06.975910 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:06.976120 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:06.976012 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:06.976120 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:06.976080 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:06.976277 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:06.976147 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:08.658743 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:08.658708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:08.659165 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:08.658760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:08.659165 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:08.658930 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:08.659165 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:08.658991 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret podName:27a5cbb8-9293-4a54-9016-a822e9633ed2 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:16.658973283 +0000 UTC m=+18.099863721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret") pod "global-pull-secret-syncer-qjr5v" (UID: "27a5cbb8-9293-4a54-9016-a822e9633ed2") : object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:08.659165 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:08.659060 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:08.659165 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:08.659100 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs podName:9f8dcf84-140a-4955-933b-6f40794f9b9e nodeName:}" failed. No retries permitted until 2026-08-02 09:33:16.659087773 +0000 UTC m=+18.099978208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs") pod "network-metrics-daemon-wr257" (UID: "9f8dcf84-140a-4955-933b-6f40794f9b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:08.759954 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:08.759924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwv4\" (UniqueName: \"kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4\") pod \"network-check-target-68nsw\" (UID: \"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30\") " pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:08.760129 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:08.760069 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 02 09:33:08.760129 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:08.760084 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 02 09:33:08.760129 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:08.760093 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bvwv4 for pod openshift-network-diagnostics/network-check-target-68nsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:08.760297 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:08.760152 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4 podName:75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:16.760140469 +0000 UTC m=+18.201030903 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvwv4" (UniqueName: "kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4") pod "network-check-target-68nsw" (UID: "75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:08.973907 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:08.973840 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:08.974066 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:08.973934 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:08.974496 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:08.974236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:08.974496 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:08.974345 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:08.974496 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:08.974389 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:08.974496 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:08.974451 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:10.975151 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:10.975115 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:10.975679 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:10.975254 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:10.975679 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:10.975330 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:10.975679 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:10.975431 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:10.975679 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:10.975545 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:10.975679 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:10.975640 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:12.972360 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:12.972331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:12.972360 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:12.972361 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:12.972822 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:12.972331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:12.972822 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:12.972482 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:12.972822 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:12.972586 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:12.972822 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:12.972643 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:14.973210 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:14.973178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:14.973652 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:14.973178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:14.973652 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:14.973319 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:14.973652 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:14.973321 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:14.973652 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:14.973419 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:14.973652 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:14.973517 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:16.723752 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:16.723718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:16.723752 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:16.723754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:16.724246 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:16.723839 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:16.724246 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:16.723878 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:16.724246 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:16.723887 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret podName:27a5cbb8-9293-4a54-9016-a822e9633ed2 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:32.723875416 +0000 UTC m=+34.164765850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret") pod "global-pull-secret-syncer-qjr5v" (UID: "27a5cbb8-9293-4a54-9016-a822e9633ed2") : object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:16.724246 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:16.723954 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs podName:9f8dcf84-140a-4955-933b-6f40794f9b9e nodeName:}" failed. No retries permitted until 2026-08-02 09:33:32.723935574 +0000 UTC m=+34.164826008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs") pod "network-metrics-daemon-wr257" (UID: "9f8dcf84-140a-4955-933b-6f40794f9b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:16.824574 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:16.824545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwv4\" (UniqueName: \"kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4\") pod \"network-check-target-68nsw\" (UID: \"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30\") " pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:16.824715 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:16.824691 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 02 09:33:16.824715 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:16.824710 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 02 09:33:16.824828 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:16.824722 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bvwv4 for pod openshift-network-diagnostics/network-check-target-68nsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:16.824828 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:16.824787 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4 podName:75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30 nodeName:}" failed. No retries permitted until 2026-08-02 09:33:32.824767635 +0000 UTC m=+34.265658084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvwv4" (UniqueName: "kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4") pod "network-check-target-68nsw" (UID: "75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:16.972436 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:16.972390 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:16.972624 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:16.972391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:16.972624 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:16.972521 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:16.972624 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:16.972517 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:16.972764 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:16.972627 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:16.972764 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:16.972732 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:18.975519 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:18.974501 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:18.975519 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:18.974774 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:18.975519 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:18.975333 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:18.975519 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:18.975477 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:18.975519 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:18.975520 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:18.976171 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:18.975695 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:20.972655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:20.972433 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:20.973408 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:20.972706 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:20.973408 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:20.972510 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:20.973408 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:20.972800 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:20.973408 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:20.972449 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:20.973408 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:20.972894 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:21.074918 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.074715 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n4td2" event={"ID":"3d42b000-3950-4ca8-b754-e38a28f50662","Type":"ContainerStarted","Data":"20a50e0ad1ddfeaeae6276f12ea9b07853e5c39868c243d69f39a78c53ed8227"} Aug 02 09:33:21.076139 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.076115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jxtkm" event={"ID":"6be5c331-cc91-4b21-b720-fbf73f0da999","Type":"ContainerStarted","Data":"7437c1a2252dc73396d1fff10d700205cbabe14a66dbb5f25770d9907256275a"} Aug 02 09:33:21.077519 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.077491 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dbgqp" event={"ID":"3310e94b-d9e4-4f11-8c91-b54387f17a5e","Type":"ContainerStarted","Data":"f6eb90029378436b70f8683f051b920339c66fab961f25b68474b6053c9ad857"} Aug 02 09:33:21.078919 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.078896 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qv7kg" event={"ID":"c29d8355-7a1f-4575-86d5-4f027d749e11","Type":"ContainerStarted","Data":"a7cdb1e37ba59571722cc6753ba55daecea85a76d030011914771340ba92a9a7"} Aug 02 09:33:21.080237 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.080212 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g46pd" event={"ID":"dc472391-a90e-4235-a348-e96b0cec389f","Type":"ContainerStarted","Data":"6aaaab2859ddead8651ae6dca6bae791edcce11f450583f5b707e198ae6842c5"} Aug 02 09:33:21.081659 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.081639 2564 generic.go:356] "Generic (PLEG): container finished" podID="0457b78e-44a9-4746-a24c-22b355904bb3" containerID="2dbdcbbbe01e2013139a06b2c92d458f617e838cec9e86b45d40003cd09138be" exitCode=0 Aug 02 09:33:21.081742 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.081717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" event={"ID":"0457b78e-44a9-4746-a24c-22b355904bb3","Type":"ContainerDied","Data":"2dbdcbbbe01e2013139a06b2c92d458f617e838cec9e86b45d40003cd09138be"} Aug 02 09:33:21.083879 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.083857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" event={"ID":"a1bb0df9-1cc4-449a-8d50-76efb8b76a43","Type":"ContainerStarted","Data":"d2a3dca0aa783d30eac3484da0d35efaa1a6002ce87ca586c487fbbfe5e8319b"} Aug 02 09:33:21.083951 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.083917 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" event={"ID":"a1bb0df9-1cc4-449a-8d50-76efb8b76a43","Type":"ContainerStarted","Data":"ed7c1326307d9fb3e165b5f9a1906c47c6bbd6237275f779f0aae3b7ff929982"} Aug 02 09:33:21.083999 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.083962 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" event={"ID":"a1bb0df9-1cc4-449a-8d50-76efb8b76a43","Type":"ContainerStarted","Data":"747896c29ccf0bdf6caaba931d03ddfddd4f8451ffbf14a563de4b73f2264afb"} Aug 02 09:33:21.085304 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.085287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" event={"ID":"4117d772-71c5-4e71-b5d1-f19a215a9e1f","Type":"ContainerStarted","Data":"7fd3151eb0080182101d03dbe252ec3f9475ceac59281f8fe0e5cca32d476871"} Aug 02 09:33:21.088561 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.088527 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" podStartSLOduration=21.088517029 podStartE2EDuration="21.088517029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:04.058110054 +0000 UTC m=+5.499000509" watchObservedRunningTime="2026-08-02 09:33:21.088517029 +0000 UTC m=+22.529407485" Aug 02 09:33:21.100704 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.100672 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n4td2" podStartSLOduration=7.30760963 podStartE2EDuration="22.100660821s" totalImagesPullingTime="14.793051191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:21.088385235 +0000 UTC m=+22.529275690" watchObservedRunningTime="2026-08-02 09:33:21.100660821 +0000 UTC m=+22.541551277" Aug 02 09:33:21.101058 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.101037 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dbgqp" podStartSLOduration=7.293528385 podStartE2EDuration="22.10102955s" totalImagesPullingTime="14.807501165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:21.100376909 +0000 UTC m=+22.541267365" watchObservedRunningTime="2026-08-02 09:33:21.10102955 +0000 UTC m=+22.541920006" Aug 02 09:33:21.121176 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.121141 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qv7kg" podStartSLOduration=3.370288726 podStartE2EDuration="22.121129373s" totalImagesPullingTime="18.750840647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:21.120577453 +0000 UTC m=+22.561467908" watchObservedRunningTime="2026-08-02 09:33:21.121129373 +0000 UTC m=+22.562019828" Aug 02 09:33:21.154800 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.154769 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g46pd" podStartSLOduration=3.436752754 podStartE2EDuration="22.154756002s" totalImagesPullingTime="18.718003248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:21.134951046 +0000 UTC m=+22.575841502" watchObservedRunningTime="2026-08-02 09:33:21.154756002 +0000 UTC m=+22.595646458" Aug 02 09:33:21.982502 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.982414 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:21.983256 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.983236 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:21.996380 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:21.996336 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jxtkm" podStartSLOduration=4.3004111720000004 podStartE2EDuration="22.996322187s" totalImagesPullingTime="18.695911015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:21.168908971 +0000 UTC m=+22.609799426" watchObservedRunningTime="2026-08-02 09:33:21.996322187 +0000 UTC m=+23.437212652" Aug 02 09:33:22.089204 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.089179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j7hkg" event={"ID":"b528680d-7978-4bba-ad55-8c68bb7bd545","Type":"ContainerStarted","Data":"f683bf5c6eb50f1c64cb476cfd98f15ca1d4e90336257e369a54c9d6e52485e1"} Aug 02 09:33:22.092381 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.092353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" event={"ID":"a1bb0df9-1cc4-449a-8d50-76efb8b76a43","Type":"ContainerStarted","Data":"c3cebaa5160fc84a4248fa20ff9eef5ec47622546b8cb727f22988ad1ce99968"} Aug 02 09:33:22.092572 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.092416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" event={"ID":"a1bb0df9-1cc4-449a-8d50-76efb8b76a43","Type":"ContainerStarted","Data":"56be6761072d8696c94a467311626ede673d128bc3cca91568f3e3b9c6b379a5"} Aug 02 09:33:22.092572 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.092456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" event={"ID":"a1bb0df9-1cc4-449a-8d50-76efb8b76a43","Type":"ContainerStarted","Data":"441bcd0944ab6fc15b161d7cc69a47d1c791fe16a29bcd7483b312e6c83a52c1"} Aug 02 09:33:22.106766 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.106712 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j7hkg" podStartSLOduration=4.409236842 podStartE2EDuration="23.106693627s" totalImagesPullingTime="18.697456785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:22.10613886 +0000 UTC m=+23.547029318" watchObservedRunningTime="2026-08-02 09:33:22.106693627 +0000 UTC m=+23.547584085" Aug 02 09:33:22.231352 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.231326 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 02 09:33:22.577148 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.577070 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:22.577757 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.577733 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n4td2" Aug 02 09:33:22.950341 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.950179 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-02T09:33:22.231348579Z","UUID":"2f005826-f6ce-422c-b870-a1b1152dab59","Handler":null,"Name":"","Endpoint":""} Aug 02 09:33:22.951905 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.951880 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 02 09:33:22.952025 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.951924 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 02 09:33:22.972479 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.972441 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:22.972479 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.972453 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:22.972652 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:22.972575 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:22.972748 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:22.972727 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:22.972818 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:22.972802 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:22.972925 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:22.972906 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:23.097284 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:23.097244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" event={"ID":"4117d772-71c5-4e71-b5d1-f19a215a9e1f","Type":"ContainerStarted","Data":"638d6650f1a75871207392be6503a87ad2480146b544ef80bd9135a60c422f07"} Aug 02 09:33:24.101812 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:24.101771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" event={"ID":"4117d772-71c5-4e71-b5d1-f19a215a9e1f","Type":"ContainerStarted","Data":"ec6ee830bbe752492ea91a0424399c3334d02baad4ef7b2ee277031f637da24f"} Aug 02 09:33:24.117671 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:24.117627 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b9ckh" podStartSLOduration=4.262191166 podStartE2EDuration="25.117614385s" totalImagesPullingTime="20.855423219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:24.11744005 +0000 UTC m=+25.558330508" watchObservedRunningTime="2026-08-02 09:33:24.117614385 +0000 UTC m=+25.558504842" Aug 02 09:33:24.972547 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:24.972516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:24.972711 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:24.972516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:24.972711 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:24.972516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:24.972711 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:24.972640 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:24.972860 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:24.972737 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:24.972860 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:24.972825 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:26.109451 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:26.109246 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" event={"ID":"a1bb0df9-1cc4-449a-8d50-76efb8b76a43","Type":"ContainerStarted","Data":"5a9e04ee853bb4dd88f58761beed20613cc121f9c216bff86acbb5e380cba8f6"} Aug 02 09:33:26.110913 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:26.110887 2564 generic.go:356] "Generic (PLEG): container finished" podID="0457b78e-44a9-4746-a24c-22b355904bb3" containerID="1ffa55e719b3e0dc4f8bdec612420a9dccea52d120172c7a24379a6db136157e" exitCode=0 Aug 02 09:33:26.111005 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:26.110918 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" event={"ID":"0457b78e-44a9-4746-a24c-22b355904bb3","Type":"ContainerDied","Data":"1ffa55e719b3e0dc4f8bdec612420a9dccea52d120172c7a24379a6db136157e"} Aug 02 09:33:26.972385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:26.972359 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:26.972558 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:26.972364 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:26.972558 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:26.972487 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:26.972558 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:26.972502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:26.972684 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:26.972577 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:26.972684 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:26.972646 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:28.116567 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:28.116285 2564 generic.go:356] "Generic (PLEG): container finished" podID="0457b78e-44a9-4746-a24c-22b355904bb3" containerID="cf2d42a5bff0a23ed7a0d8ef5b00bdf66a30fb6eb803c044a6853aa8e0470605" exitCode=0 Aug 02 09:33:28.117390 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:28.116363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" event={"ID":"0457b78e-44a9-4746-a24c-22b355904bb3","Type":"ContainerDied","Data":"cf2d42a5bff0a23ed7a0d8ef5b00bdf66a30fb6eb803c044a6853aa8e0470605"} Aug 02 09:33:28.120586 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:28.120563 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" event={"ID":"a1bb0df9-1cc4-449a-8d50-76efb8b76a43","Type":"ContainerStarted","Data":"ea9e645dfbd70bb059fae7a9802b0188ea8f5b5617357980b4204ea6bc39c95d"} Aug 02 09:33:28.120882 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:28.120864 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:28.120978 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:28.120946 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:28.136320 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:28.136300 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:28.163019 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:28.161103 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" podStartSLOduration=10.392679206 podStartE2EDuration="29.161085104s" totalImagesPullingTime="18.768405898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:28.159584397 +0000 UTC m=+29.600474853" watchObservedRunningTime="2026-08-02 09:33:28.161085104 +0000 UTC m=+29.601975561" Aug 02 09:33:28.972870 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:28.972833 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:28.973090 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:28.972908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:28.973090 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:28.972915 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:28.973090 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:28.973021 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:28.973090 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:28.973056 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:28.973273 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:28.973169 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:29.125255 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:29.125218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" event={"ID":"0457b78e-44a9-4746-a24c-22b355904bb3","Type":"ContainerStarted","Data":"ec4909eb5606fc0462d94e83d8b367fb2d7d6a014f6a61d6edec1735d30fdca3"} Aug 02 09:33:29.125821 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:29.125788 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:29.141686 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:29.141659 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:33:29.302675 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:29.302600 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qjr5v"] Aug 02 09:33:29.302813 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:29.302727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:29.302893 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:29.302873 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:29.303540 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:29.303518 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-68nsw"] Aug 02 09:33:29.303653 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:29.303587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:29.303713 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:29.303672 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:29.317497 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:29.317457 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wr257"] Aug 02 09:33:29.317622 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:29.317560 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:29.317677 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:29.317652 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:30.129919 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:30.129881 2564 generic.go:356] "Generic (PLEG): container finished" podID="0457b78e-44a9-4746-a24c-22b355904bb3" containerID="ec4909eb5606fc0462d94e83d8b367fb2d7d6a014f6a61d6edec1735d30fdca3" exitCode=0 Aug 02 09:33:30.130324 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:30.129962 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" event={"ID":"0457b78e-44a9-4746-a24c-22b355904bb3","Type":"ContainerDied","Data":"ec4909eb5606fc0462d94e83d8b367fb2d7d6a014f6a61d6edec1735d30fdca3"} Aug 02 09:33:30.973272 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:30.973204 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:30.973272 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:30.973233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:30.973455 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:30.973341 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:30.973455 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:30.973394 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:30.973581 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:30.973486 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:30.973685 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:30.973660 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:31.166281 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:31.166255 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dbgqp_3310e94b-d9e4-4f11-8c91-b54387f17a5e/dns-node-resolver/0.log" Aug 02 09:33:32.147501 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:32.147253 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jxtkm_6be5c331-cc91-4b21-b720-fbf73f0da999/node-ca/0.log" Aug 02 09:33:32.742000 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:32.741965 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:32.742539 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:32.742017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:32.742539 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:32.742153 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:32.742539 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:32.742230 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs podName:9f8dcf84-140a-4955-933b-6f40794f9b9e nodeName:}" failed. No retries permitted until 2026-08-02 09:34:04.742206652 +0000 UTC m=+66.183097100 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs") pod "network-metrics-daemon-wr257" (UID: "9f8dcf84-140a-4955-933b-6f40794f9b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 02 09:33:32.742539 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:32.742157 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:32.742539 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:32.742312 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret podName:27a5cbb8-9293-4a54-9016-a822e9633ed2 nodeName:}" failed. No retries permitted until 2026-08-02 09:34:04.742287948 +0000 UTC m=+66.183178387 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret") pod "global-pull-secret-syncer-qjr5v" (UID: "27a5cbb8-9293-4a54-9016-a822e9633ed2") : object "kube-system"/"original-pull-secret" not registered Aug 02 09:33:32.842613 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:32.842578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwv4\" (UniqueName: \"kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4\") pod \"network-check-target-68nsw\" (UID: \"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30\") " pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:32.842793 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:32.842714 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 02 09:33:32.842793 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:32.842730 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 02 09:33:32.842793 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:32.842740 2564 projected.go:196] Error preparing data for projected volume kube-api-access-bvwv4 for pod openshift-network-diagnostics/network-check-target-68nsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:32.842793 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:32.842790 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4 podName:75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30 nodeName:}" failed. No retries permitted until 2026-08-02 09:34:04.842771378 +0000 UTC m=+66.283661813 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bvwv4" (UniqueName: "kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4") pod "network-check-target-68nsw" (UID: "75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 02 09:33:32.972698 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:32.972667 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:32.972698 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:32.972694 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:32.972916 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:32.972668 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:32.972916 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:32.972785 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-68nsw" podUID="75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30" Aug 02 09:33:32.972916 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:32.972899 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wr257" podUID="9f8dcf84-140a-4955-933b-6f40794f9b9e" Aug 02 09:33:32.973038 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:33:32.972996 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjr5v" podUID="27a5cbb8-9293-4a54-9016-a822e9633ed2" Aug 02 09:33:33.446980 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:33.446948 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeReady" Aug 02 09:33:33.447130 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:33.447062 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 02 09:33:34.972784 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:34.972743 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:33:34.973313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:34.972742 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:33:34.973313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:34.972743 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:33:34.976012 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:34.975990 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 02 09:33:34.976012 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:34.976005 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 02 09:33:34.976642 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:34.976623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 02 09:33:34.976642 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:34.976642 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 02 09:33:34.976938 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:34.976922 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt8nz\"" Aug 02 09:33:34.977000 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:34.976984 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mbl9n\"" Aug 02 09:33:35.144714 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:35.144684 2564 generic.go:356] "Generic (PLEG): container finished" podID="0457b78e-44a9-4746-a24c-22b355904bb3" containerID="988c93d4bb5de95212e91e7db27cbdd80bb75dd2e1200c0eea7d9ce7798f3334" exitCode=0 Aug 02 09:33:35.144817 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:35.144733 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" event={"ID":"0457b78e-44a9-4746-a24c-22b355904bb3","Type":"ContainerDied","Data":"988c93d4bb5de95212e91e7db27cbdd80bb75dd2e1200c0eea7d9ce7798f3334"} Aug 02 09:33:36.149997 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:36.149965 2564 generic.go:356] "Generic (PLEG): container finished" podID="0457b78e-44a9-4746-a24c-22b355904bb3" containerID="d892b39f46d157b22f47a35cd688c5cc2a8bf79f8e3c1d97b6ff658fa83bb388" exitCode=0 Aug 02 09:33:36.149997 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:36.149999 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" event={"ID":"0457b78e-44a9-4746-a24c-22b355904bb3","Type":"ContainerDied","Data":"d892b39f46d157b22f47a35cd688c5cc2a8bf79f8e3c1d97b6ff658fa83bb388"} Aug 02 09:33:37.154972 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:37.154933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" event={"ID":"0457b78e-44a9-4746-a24c-22b355904bb3","Type":"ContainerStarted","Data":"62b0bcdda511a5b4999925ed86031949a58d4aab1318eafd17e6c724478226a9"} Aug 02 09:33:37.177283 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:37.177237 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4zlt5" podStartSLOduration=9.440835559 podStartE2EDuration="38.177221367s" totalImagesPullingTime="28.736385808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:37.176169099 +0000 UTC m=+38.617059557" watchObservedRunningTime="2026-08-02 09:33:37.177221367 +0000 UTC m=+38.618111822" Aug 02 09:33:50.949027 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.948995 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sxvlb"] Aug 02 09:33:50.962395 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.962368 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mdqdc"] Aug 02 09:33:50.962611 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.962532 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sxvlb" Aug 02 09:33:50.965015 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.964988 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 02 09:33:50.965317 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.965278 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 02 09:33:50.965405 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.965341 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4s9vb\"" Aug 02 09:33:50.965494 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.965477 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 02 09:33:50.976585 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.976557 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:50.978868 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.978845 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 02 09:33:50.979228 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.979057 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zv4ml\"" Aug 02 09:33:50.979228 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.979216 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 02 09:33:50.980902 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:50.980882 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c7cbb5c7f-q759r"] Aug 02 09:33:51.002384 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.002367 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sxvlb"] Aug 02 09:33:51.002492 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.002392 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mdqdc"] Aug 02 09:33:51.002492 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.002408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c7cbb5c7f-q759r"] Aug 02 09:33:51.002492 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.002482 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.005397 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.005378 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 02 09:33:51.005506 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.005415 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 02 09:33:51.005714 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.005697 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 02 09:33:51.005773 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.005761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f75jg\"" Aug 02 09:33:51.010086 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.010068 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 02 09:33:51.051477 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.051444 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c2stw"] Aug 02 09:33:51.074437 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.074421 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2stw"] Aug 02 09:33:51.074539 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.074528 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.076915 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.076888 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 02 09:33:51.077028 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.076981 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 02 09:33:51.077118 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.076984 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 02 09:33:51.077118 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.077045 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-49fv9\"" Aug 02 09:33:51.077240 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.077120 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 02 09:33:51.158379 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1bff3366-0670-4bc9-8327-a326370b731a-installation-pull-secrets\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.158494 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158392 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bff3366-0670-4bc9-8327-a326370b731a-bound-sa-token\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.158494 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9028085f-6881-4a18-92b3-343810822d8d-cert\") pod \"ingress-canary-sxvlb\" (UID: \"9028085f-6881-4a18-92b3-343810822d8d\") " pod="openshift-ingress-canary/ingress-canary-sxvlb" Aug 02 09:33:51.158494 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1bff3366-0670-4bc9-8327-a326370b731a-trusted-ca\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.158600 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1bff3366-0670-4bc9-8327-a326370b731a-image-registry-private-configuration\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.158600 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158516 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1bff3366-0670-4bc9-8327-a326370b731a-registry-certificates\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.158600 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f633b7b6-33dd-492f-8cd0-1a6604102b82-metrics-tls\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.158600 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158559 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1bff3366-0670-4bc9-8327-a326370b731a-registry-tls\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.158600 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmj5v\" (UniqueName: \"kubernetes.io/projected/9028085f-6881-4a18-92b3-343810822d8d-kube-api-access-hmj5v\") pod \"ingress-canary-sxvlb\" (UID: \"9028085f-6881-4a18-92b3-343810822d8d\") " pod="openshift-ingress-canary/ingress-canary-sxvlb" Aug 02 09:33:51.158600 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158596 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f633b7b6-33dd-492f-8cd0-1a6604102b82-tmp-dir\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.158770 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7lcv\" (UniqueName: \"kubernetes.io/projected/f633b7b6-33dd-492f-8cd0-1a6604102b82-kube-api-access-c7lcv\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.158770 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158679 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tnbh\" (UniqueName: \"kubernetes.io/projected/1bff3366-0670-4bc9-8327-a326370b731a-kube-api-access-7tnbh\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.158770 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1bff3366-0670-4bc9-8327-a326370b731a-ca-trust-extracted\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.158770 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.158720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f633b7b6-33dd-492f-8cd0-1a6604102b82-config-volume\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.259396 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1bff3366-0670-4bc9-8327-a326370b731a-ca-trust-extracted\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.259396 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f633b7b6-33dd-492f-8cd0-1a6604102b82-config-volume\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.259549 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259415 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1bff3366-0670-4bc9-8327-a326370b731a-installation-pull-secrets\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.259592 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bff3366-0670-4bc9-8327-a326370b731a-bound-sa-token\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.259592 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259578 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29kx\" (UniqueName: \"kubernetes.io/projected/5c4288d5-ac54-473a-88b4-ce6df5d95b96-kube-api-access-d29kx\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.259681 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9028085f-6881-4a18-92b3-343810822d8d-cert\") pod \"ingress-canary-sxvlb\" (UID: \"9028085f-6881-4a18-92b3-343810822d8d\") " pod="openshift-ingress-canary/ingress-canary-sxvlb" Aug 02 09:33:51.259681 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1bff3366-0670-4bc9-8327-a326370b731a-trusted-ca\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.259769 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1bff3366-0670-4bc9-8327-a326370b731a-image-registry-private-configuration\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.259825 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c4288d5-ac54-473a-88b4-ce6df5d95b96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.259925 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1bff3366-0670-4bc9-8327-a326370b731a-registry-certificates\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.259987 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f633b7b6-33dd-492f-8cd0-1a6604102b82-metrics-tls\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.260039 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.259981 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f633b7b6-33dd-492f-8cd0-1a6604102b82-config-volume\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.260039 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1bff3366-0670-4bc9-8327-a326370b731a-registry-tls\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.260039 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260016 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1bff3366-0670-4bc9-8327-a326370b731a-ca-trust-extracted\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.260039 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260035 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c4288d5-ac54-473a-88b4-ce6df5d95b96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.260228 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmj5v\" (UniqueName: \"kubernetes.io/projected/9028085f-6881-4a18-92b3-343810822d8d-kube-api-access-hmj5v\") pod \"ingress-canary-sxvlb\" (UID: \"9028085f-6881-4a18-92b3-343810822d8d\") " pod="openshift-ingress-canary/ingress-canary-sxvlb" Aug 02 09:33:51.260228 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f633b7b6-33dd-492f-8cd0-1a6604102b82-tmp-dir\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.260228 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7lcv\" (UniqueName: \"kubernetes.io/projected/f633b7b6-33dd-492f-8cd0-1a6604102b82-kube-api-access-c7lcv\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.260228 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c4288d5-ac54-473a-88b4-ce6df5d95b96-crio-socket\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.260434 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tnbh\" (UniqueName: \"kubernetes.io/projected/1bff3366-0670-4bc9-8327-a326370b731a-kube-api-access-7tnbh\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.260434 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260262 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c4288d5-ac54-473a-88b4-ce6df5d95b96-data-volume\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.260930 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260711 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1bff3366-0670-4bc9-8327-a326370b731a-registry-certificates\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.260930 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1bff3366-0670-4bc9-8327-a326370b731a-trusted-ca\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.260930 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.260770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f633b7b6-33dd-492f-8cd0-1a6604102b82-tmp-dir\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.263691 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.263668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f633b7b6-33dd-492f-8cd0-1a6604102b82-metrics-tls\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.263823 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.263804 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1bff3366-0670-4bc9-8327-a326370b731a-registry-tls\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.263879 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.263822 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1bff3366-0670-4bc9-8327-a326370b731a-image-registry-private-configuration\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.263954 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.263932 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1bff3366-0670-4bc9-8327-a326370b731a-installation-pull-secrets\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.263985 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.263962 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9028085f-6881-4a18-92b3-343810822d8d-cert\") pod \"ingress-canary-sxvlb\" (UID: \"9028085f-6881-4a18-92b3-343810822d8d\") " pod="openshift-ingress-canary/ingress-canary-sxvlb" Aug 02 09:33:51.269195 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.269169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tnbh\" (UniqueName: \"kubernetes.io/projected/1bff3366-0670-4bc9-8327-a326370b731a-kube-api-access-7tnbh\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.269312 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.269294 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmj5v\" (UniqueName: \"kubernetes.io/projected/9028085f-6881-4a18-92b3-343810822d8d-kube-api-access-hmj5v\") pod \"ingress-canary-sxvlb\" (UID: \"9028085f-6881-4a18-92b3-343810822d8d\") " pod="openshift-ingress-canary/ingress-canary-sxvlb" Aug 02 09:33:51.269372 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.269348 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bff3366-0670-4bc9-8327-a326370b731a-bound-sa-token\") pod \"image-registry-c7cbb5c7f-q759r\" (UID: \"1bff3366-0670-4bc9-8327-a326370b731a\") " pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.269431 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.269392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7lcv\" (UniqueName: \"kubernetes.io/projected/f633b7b6-33dd-492f-8cd0-1a6604102b82-kube-api-access-c7lcv\") pod \"dns-default-mdqdc\" (UID: \"f633b7b6-33dd-492f-8cd0-1a6604102b82\") " pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.273277 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.273259 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sxvlb" Aug 02 09:33:51.287957 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.287938 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:51.310743 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.310724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:51.361979 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.361190 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c4288d5-ac54-473a-88b4-ce6df5d95b96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.361979 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.361240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c4288d5-ac54-473a-88b4-ce6df5d95b96-crio-socket\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.361979 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.361266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c4288d5-ac54-473a-88b4-ce6df5d95b96-data-volume\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.361979 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.361321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29kx\" (UniqueName: \"kubernetes.io/projected/5c4288d5-ac54-473a-88b4-ce6df5d95b96-kube-api-access-d29kx\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.361979 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.361369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c4288d5-ac54-473a-88b4-ce6df5d95b96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.361979 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.361718 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c4288d5-ac54-473a-88b4-ce6df5d95b96-crio-socket\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.361979 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.361896 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c4288d5-ac54-473a-88b4-ce6df5d95b96-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.362900 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.362252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c4288d5-ac54-473a-88b4-ce6df5d95b96-data-volume\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.364085 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.364066 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c4288d5-ac54-473a-88b4-ce6df5d95b96-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.368489 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.368453 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29kx\" (UniqueName: \"kubernetes.io/projected/5c4288d5-ac54-473a-88b4-ce6df5d95b96-kube-api-access-d29kx\") pod \"insights-runtime-extractor-c2stw\" (UID: \"5c4288d5-ac54-473a-88b4-ce6df5d95b96\") " pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.382384 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.382362 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2stw" Aug 02 09:33:51.493970 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.493938 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sxvlb"] Aug 02 09:33:51.497044 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:33:51.497016 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9028085f_6881_4a18_92b3_343810822d8d.slice/crio-2bd04057a72b7707f10f303657c79510e0da4033561f2caf9a3921d97acb5154 WatchSource:0}: Error finding container 2bd04057a72b7707f10f303657c79510e0da4033561f2caf9a3921d97acb5154: Status 404 returned error can't find the container with id 2bd04057a72b7707f10f303657c79510e0da4033561f2caf9a3921d97acb5154 Aug 02 09:33:51.553270 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.553159 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2stw"] Aug 02 09:33:51.555587 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:33:51.555557 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4288d5_ac54_473a_88b4_ce6df5d95b96.slice/crio-6992287e6ddc0dda1f00e7abb78e24bc082a2073fb411956ed47ba363bd556be WatchSource:0}: Error finding container 6992287e6ddc0dda1f00e7abb78e24bc082a2073fb411956ed47ba363bd556be: Status 404 returned error can't find the container with id 6992287e6ddc0dda1f00e7abb78e24bc082a2073fb411956ed47ba363bd556be Aug 02 09:33:51.707892 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.707866 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mdqdc"] Aug 02 09:33:51.710782 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:33:51.710724 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf633b7b6_33dd_492f_8cd0_1a6604102b82.slice/crio-2baa033999e906791a99eb874c7b981d93e63d59bebdb87aba71d50163d8db73 WatchSource:0}: Error finding container 2baa033999e906791a99eb874c7b981d93e63d59bebdb87aba71d50163d8db73: Status 404 returned error can't find the container with id 2baa033999e906791a99eb874c7b981d93e63d59bebdb87aba71d50163d8db73 Aug 02 09:33:51.710915 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:51.710895 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c7cbb5c7f-q759r"] Aug 02 09:33:51.713383 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:33:51.713357 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bff3366_0670_4bc9_8327_a326370b731a.slice/crio-6647f0837be2112cbe1891702aded683a6bd026f8dc99f2fceac45c4ed05cbe4 WatchSource:0}: Error finding container 6647f0837be2112cbe1891702aded683a6bd026f8dc99f2fceac45c4ed05cbe4: Status 404 returned error can't find the container with id 6647f0837be2112cbe1891702aded683a6bd026f8dc99f2fceac45c4ed05cbe4 Aug 02 09:33:52.189762 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:52.189684 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mdqdc" event={"ID":"f633b7b6-33dd-492f-8cd0-1a6604102b82","Type":"ContainerStarted","Data":"2baa033999e906791a99eb874c7b981d93e63d59bebdb87aba71d50163d8db73"} Aug 02 09:33:52.191608 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:52.191559 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2stw" event={"ID":"5c4288d5-ac54-473a-88b4-ce6df5d95b96","Type":"ContainerStarted","Data":"daa438e222c13aa704e246f8b224e243fe99359c40c918fb1816bb983fdde178"} Aug 02 09:33:52.191608 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:52.191612 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2stw" event={"ID":"5c4288d5-ac54-473a-88b4-ce6df5d95b96","Type":"ContainerStarted","Data":"6992287e6ddc0dda1f00e7abb78e24bc082a2073fb411956ed47ba363bd556be"} Aug 02 09:33:52.193146 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:52.193089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sxvlb" event={"ID":"9028085f-6881-4a18-92b3-343810822d8d","Type":"ContainerStarted","Data":"2bd04057a72b7707f10f303657c79510e0da4033561f2caf9a3921d97acb5154"} Aug 02 09:33:52.195736 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:52.195708 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" event={"ID":"1bff3366-0670-4bc9-8327-a326370b731a","Type":"ContainerStarted","Data":"389030623ab74c55a188ab0f3f98eaa3962b19d63ad78a70d3e5f52ecb01ba5b"} Aug 02 09:33:52.195841 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:52.195759 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" event={"ID":"1bff3366-0670-4bc9-8327-a326370b731a","Type":"ContainerStarted","Data":"6647f0837be2112cbe1891702aded683a6bd026f8dc99f2fceac45c4ed05cbe4"} Aug 02 09:33:52.196351 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:52.196301 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:33:52.219953 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:52.215818 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" podStartSLOduration=2.215801044 podStartE2EDuration="2.215801044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:52.213420074 +0000 UTC m=+53.654310532" watchObservedRunningTime="2026-08-02 09:33:52.215801044 +0000 UTC m=+53.656691500" Aug 02 09:33:53.201842 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:53.201805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2stw" event={"ID":"5c4288d5-ac54-473a-88b4-ce6df5d95b96","Type":"ContainerStarted","Data":"d06be003d996595e576f033dc813f085e88baceb5500dd1dfd103c670ce52bcc"} Aug 02 09:33:55.210891 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:55.210858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mdqdc" event={"ID":"f633b7b6-33dd-492f-8cd0-1a6604102b82","Type":"ContainerStarted","Data":"6d7d08f05e8fe0743c7ea34eb1481ee6d4c8850a7b30f04b3b166a5df0e327ae"} Aug 02 09:33:55.211307 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:55.210914 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mdqdc" event={"ID":"f633b7b6-33dd-492f-8cd0-1a6604102b82","Type":"ContainerStarted","Data":"0ae94d3229855a1a9fcd9927c920f76b15cd7d6d7e47185ab9af8ce5ffc875ec"} Aug 02 09:33:55.211307 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:55.210938 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mdqdc" Aug 02 09:33:55.212845 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:55.212817 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2stw" event={"ID":"5c4288d5-ac54-473a-88b4-ce6df5d95b96","Type":"ContainerStarted","Data":"51bfff7d8837286582028e6ce14ff2a2d6968aed20a9de15f744bb076ab2d803"} Aug 02 09:33:55.214356 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:55.214334 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sxvlb" event={"ID":"9028085f-6881-4a18-92b3-343810822d8d","Type":"ContainerStarted","Data":"9e66f58fa5d09f7e06a72a4404146466f397bff8e763f8e5ac7bb4d14bc615cd"} Aug 02 09:33:55.225376 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:55.225333 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mdqdc" podStartSLOduration=2.457088879 podStartE2EDuration="5.225319576s" totalImagesPullingTime="2.768230697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:55.225018114 +0000 UTC m=+56.665908572" watchObservedRunningTime="2026-08-02 09:33:55.225319576 +0000 UTC m=+56.666210010" Aug 02 09:33:55.241664 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:55.241619 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sxvlb" podStartSLOduration=2.253577522 podStartE2EDuration="5.241606919s" totalImagesPullingTime="2.988029397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:55.240863235 +0000 UTC m=+56.681753685" watchObservedRunningTime="2026-08-02 09:33:55.241606919 +0000 UTC m=+56.682497410" Aug 02 09:33:55.259150 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:55.256858 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c2stw" podStartSLOduration=1.529390273 podStartE2EDuration="4.256841154s" totalImagesPullingTime="2.727450881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:33:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:55.256095021 +0000 UTC m=+56.696985479" watchObservedRunningTime="2026-08-02 09:33:55.256841154 +0000 UTC m=+56.697731611" Aug 02 09:33:56.814136 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:56.814106 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77"] Aug 02 09:33:56.815855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:56.815839 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77" Aug 02 09:33:56.817841 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:56.817816 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-msk7f\"" Aug 02 09:33:56.817963 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:56.817818 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Aug 02 09:33:56.826323 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:56.826303 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77"] Aug 02 09:33:56.904900 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:56.904866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/abe2537e-513b-4ae4-9ae0-8632d800488a-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-ffr77\" (UID: \"abe2537e-513b-4ae4-9ae0-8632d800488a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77" Aug 02 09:33:57.005441 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:57.005405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/abe2537e-513b-4ae4-9ae0-8632d800488a-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-ffr77\" (UID: \"abe2537e-513b-4ae4-9ae0-8632d800488a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77" Aug 02 09:33:57.007788 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:57.007769 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/abe2537e-513b-4ae4-9ae0-8632d800488a-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-ffr77\" (UID: \"abe2537e-513b-4ae4-9ae0-8632d800488a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77" Aug 02 09:33:57.123861 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:57.123774 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77" Aug 02 09:33:57.236417 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:57.236388 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77"] Aug 02 09:33:57.239629 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:33:57.239599 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe2537e_513b_4ae4_9ae0_8632d800488a.slice/crio-db9572559245de7a7b0738a216bbbf6368ff194f3eae20eb0d368dac98639e04 WatchSource:0}: Error finding container db9572559245de7a7b0738a216bbbf6368ff194f3eae20eb0d368dac98639e04: Status 404 returned error can't find the container with id db9572559245de7a7b0738a216bbbf6368ff194f3eae20eb0d368dac98639e04 Aug 02 09:33:58.226440 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:58.226393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77" event={"ID":"abe2537e-513b-4ae4-9ae0-8632d800488a","Type":"ContainerStarted","Data":"db9572559245de7a7b0738a216bbbf6368ff194f3eae20eb0d368dac98639e04"} Aug 02 09:33:59.231043 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.231010 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77" event={"ID":"abe2537e-513b-4ae4-9ae0-8632d800488a","Type":"ContainerStarted","Data":"0df437ffe92636a834c1c6d4a245cbf97dd0eaf15b48ec676fec64a10c86c149"} Aug 02 09:33:59.231396 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.231191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77" Aug 02 09:33:59.235593 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.235573 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77" Aug 02 09:33:59.246139 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.246105 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-ffr77" podStartSLOduration=1.7423970020000001 podStartE2EDuration="3.246094387s" totalImagesPullingTime="1.503697385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:33:59.245202509 +0000 UTC m=+60.686092966" watchObservedRunningTime="2026-08-02 09:33:59.246094387 +0000 UTC m=+60.686984843" Aug 02 09:33:59.862770 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.862735 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-8fvdr"] Aug 02 09:33:59.865071 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.865055 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:33:59.867342 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.867260 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Aug 02 09:33:59.867342 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.867311 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 02 09:33:59.868262 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.868243 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Aug 02 09:33:59.868350 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.868267 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 02 09:33:59.868350 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.868270 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wwpvl\"" Aug 02 09:33:59.868350 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.868295 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 02 09:33:59.873576 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.873551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-8fvdr"] Aug 02 09:33:59.926009 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.925979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4ff2e2c-ba72-49e2-b161-e11788ee3560-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:33:59.926108 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.926051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4ff2e2c-ba72-49e2-b161-e11788ee3560-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:33:59.926108 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.926068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55nr2\" (UniqueName: \"kubernetes.io/projected/a4ff2e2c-ba72-49e2-b161-e11788ee3560-kube-api-access-55nr2\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:33:59.926108 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:33:59.926085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ff2e2c-ba72-49e2-b161-e11788ee3560-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:34:00.027083 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:00.027058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4ff2e2c-ba72-49e2-b161-e11788ee3560-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:34:00.027176 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:00.027099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55nr2\" (UniqueName: \"kubernetes.io/projected/a4ff2e2c-ba72-49e2-b161-e11788ee3560-kube-api-access-55nr2\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:34:00.027176 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:00.027138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ff2e2c-ba72-49e2-b161-e11788ee3560-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:34:00.027286 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:00.027176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4ff2e2c-ba72-49e2-b161-e11788ee3560-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:34:00.027401 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:34:00.027291 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Aug 02 09:34:00.027401 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:34:00.027354 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ff2e2c-ba72-49e2-b161-e11788ee3560-prometheus-operator-tls podName:a4ff2e2c-ba72-49e2-b161-e11788ee3560 nodeName:}" failed. No retries permitted until 2026-08-02 09:34:00.527339739 +0000 UTC m=+61.968230176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a4ff2e2c-ba72-49e2-b161-e11788ee3560-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-8fvdr" (UID: "a4ff2e2c-ba72-49e2-b161-e11788ee3560") : secret "prometheus-operator-tls" not found Aug 02 09:34:00.027777 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:00.027757 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4ff2e2c-ba72-49e2-b161-e11788ee3560-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:34:00.029405 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:00.029384 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4ff2e2c-ba72-49e2-b161-e11788ee3560-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:34:00.036836 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:00.036812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55nr2\" (UniqueName: \"kubernetes.io/projected/a4ff2e2c-ba72-49e2-b161-e11788ee3560-kube-api-access-55nr2\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:34:00.530005 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:00.529967 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ff2e2c-ba72-49e2-b161-e11788ee3560-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:34:00.532233 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:00.532209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4ff2e2c-ba72-49e2-b161-e11788ee3560-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-8fvdr\" (UID: \"a4ff2e2c-ba72-49e2-b161-e11788ee3560\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:34:00.773650 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:00.773615 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" Aug 02 09:34:00.890171 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:00.890089 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-8fvdr"] Aug 02 09:34:00.892546 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:34:00.892516 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ff2e2c_ba72_49e2_b161_e11788ee3560.slice/crio-78a5d889ec7db2fea65427c44bf02b36d150893417086ac165d132cceb7ed7b1 WatchSource:0}: Error finding container 78a5d889ec7db2fea65427c44bf02b36d150893417086ac165d132cceb7ed7b1: Status 404 returned error can't find the container with id 78a5d889ec7db2fea65427c44bf02b36d150893417086ac165d132cceb7ed7b1 Aug 02 09:34:01.142226 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:01.142156 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vhs2" Aug 02 09:34:01.237930 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:01.237899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" event={"ID":"a4ff2e2c-ba72-49e2-b161-e11788ee3560","Type":"ContainerStarted","Data":"78a5d889ec7db2fea65427c44bf02b36d150893417086ac165d132cceb7ed7b1"} Aug 02 09:34:02.244383 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:02.244042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" event={"ID":"a4ff2e2c-ba72-49e2-b161-e11788ee3560","Type":"ContainerStarted","Data":"dc2aa8b2477810b8bac0bf7c216926bf3b1970fdf046f8fdee2ce51185126b0f"} Aug 02 09:34:02.244383 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:02.244109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" event={"ID":"a4ff2e2c-ba72-49e2-b161-e11788ee3560","Type":"ContainerStarted","Data":"155dd28d2be3aa2643d2336bf78f82371983ec21b08de6602120f48915de8c9c"} Aug 02 09:34:04.204029 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.203975 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-8fvdr" podStartSLOduration=4.018510107 podStartE2EDuration="5.20395783s" totalImagesPullingTime="1.185447723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:02.264163042 +0000 UTC m=+63.705053498" watchObservedRunningTime="2026-08-02 09:34:04.20395783 +0000 UTC m=+65.644848285" Aug 02 09:34:04.204487 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.204372 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5"] Aug 02 09:34:04.207594 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.207577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.210014 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.209991 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Aug 02 09:34:04.210203 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.210186 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Aug 02 09:34:04.210323 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.210303 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xv5q8\"" Aug 02 09:34:04.217329 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.217309 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5"] Aug 02 09:34:04.219325 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.219306 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5kpkq"] Aug 02 09:34:04.222246 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.222229 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.224510 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.224490 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 02 09:34:04.224729 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.224710 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 02 09:34:04.224938 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.224919 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vw8xp\"" Aug 02 09:34:04.225038 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.224840 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 02 09:34:04.225952 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.225934 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph"] Aug 02 09:34:04.228968 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.228949 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.231514 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.231495 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Aug 02 09:34:04.231914 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.231554 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Aug 02 09:34:04.234432 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.231552 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-l85tn\"" Aug 02 09:34:04.234548 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.231608 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Aug 02 09:34:04.239852 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.239835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph"] Aug 02 09:34:04.255162 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255142 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-wtmp\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.255260 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq77w\" (UniqueName: \"kubernetes.io/projected/47487674-5e42-4214-a3e9-59ebf2d12c7b-kube-api-access-zq77w\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.255260 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.255345 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/073c08eb-a4a8-490f-8af8-6453451cf4dd-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.255345 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.255427 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/073c08eb-a4a8-490f-8af8-6453451cf4dd-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.255427 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz85k\" (UniqueName: \"kubernetes.io/projected/2efa155f-3829-42ea-8166-e046d9e61392-kube-api-access-bz85k\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.255427 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-textfile\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.255564 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.255564 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.255564 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2efa155f-3829-42ea-8166-e046d9e61392-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.255564 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47487674-5e42-4214-a3e9-59ebf2d12c7b-root\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.255564 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255560 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2efa155f-3829-42ea-8166-e046d9e61392-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.255802 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47487674-5e42-4214-a3e9-59ebf2d12c7b-metrics-client-ca\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.255802 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.255802 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-tls\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.255802 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2efa155f-3829-42ea-8166-e046d9e61392-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.255802 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh2v7\" (UniqueName: \"kubernetes.io/projected/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-api-access-dh2v7\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.255802 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.255734 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47487674-5e42-4214-a3e9-59ebf2d12c7b-sys\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.356614 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2efa155f-3829-42ea-8166-e046d9e61392-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.356614 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47487674-5e42-4214-a3e9-59ebf2d12c7b-root\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.356855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2efa155f-3829-42ea-8166-e046d9e61392-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.356855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356694 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47487674-5e42-4214-a3e9-59ebf2d12c7b-root\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.356855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47487674-5e42-4214-a3e9-59ebf2d12c7b-metrics-client-ca\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.356855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.356855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-tls\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.356855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2efa155f-3829-42ea-8166-e046d9e61392-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.356855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh2v7\" (UniqueName: \"kubernetes.io/projected/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-api-access-dh2v7\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.357209 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47487674-5e42-4214-a3e9-59ebf2d12c7b-sys\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.357209 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-wtmp\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.357209 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.356906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq77w\" (UniqueName: \"kubernetes.io/projected/47487674-5e42-4214-a3e9-59ebf2d12c7b-kube-api-access-zq77w\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.357209 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.357209 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/073c08eb-a4a8-490f-8af8-6453451cf4dd-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.357209 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.357209 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/073c08eb-a4a8-490f-8af8-6453451cf4dd-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.357579 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz85k\" (UniqueName: \"kubernetes.io/projected/2efa155f-3829-42ea-8166-e046d9e61392-kube-api-access-bz85k\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.357579 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357274 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-textfile\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.357579 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.357579 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.357579 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357386 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.357579 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357420 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47487674-5e42-4214-a3e9-59ebf2d12c7b-metrics-client-ca\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.357885 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47487674-5e42-4214-a3e9-59ebf2d12c7b-sys\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.357976 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.357950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/073c08eb-a4a8-490f-8af8-6453451cf4dd-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.358130 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.358107 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-textfile\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.358191 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.358173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-wtmp\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.358267 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:34:04.358251 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Aug 02 09:34:04.358347 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:34:04.358334 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-tls podName:073c08eb-a4a8-490f-8af8-6453451cf4dd nodeName:}" failed. No retries permitted until 2026-08-02 09:34:04.85831435 +0000 UTC m=+66.299204785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-fv4ph" (UID: "073c08eb-a4a8-490f-8af8-6453451cf4dd") : secret "kube-state-metrics-tls" not found Aug 02 09:34:04.358412 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.358372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.358965 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.358935 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2efa155f-3829-42ea-8166-e046d9e61392-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.359210 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.359178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/073c08eb-a4a8-490f-8af8-6453451cf4dd-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.359581 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.359560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2efa155f-3829-42ea-8166-e046d9e61392-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.359667 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.359649 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-tls\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.359854 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.359834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2efa155f-3829-42ea-8166-e046d9e61392-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.359893 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.359834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.360355 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.360333 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47487674-5e42-4214-a3e9-59ebf2d12c7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.364307 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.364289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh2v7\" (UniqueName: \"kubernetes.io/projected/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-api-access-dh2v7\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.364590 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.364572 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz85k\" (UniqueName: \"kubernetes.io/projected/2efa155f-3829-42ea-8166-e046d9e61392-kube-api-access-bz85k\") pod \"openshift-state-metrics-6bb4d7b4dd-dzcm5\" (UID: \"2efa155f-3829-42ea-8166-e046d9e61392\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.364910 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.364893 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq77w\" (UniqueName: \"kubernetes.io/projected/47487674-5e42-4214-a3e9-59ebf2d12c7b-kube-api-access-zq77w\") pod \"node-exporter-5kpkq\" (UID: \"47487674-5e42-4214-a3e9-59ebf2d12c7b\") " pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.516730 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.516708 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" Aug 02 09:34:04.530982 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.530962 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5kpkq" Aug 02 09:34:04.540936 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:34:04.540910 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47487674_5e42_4214_a3e9_59ebf2d12c7b.slice/crio-bbc52bdadca886c13d91b9f1a26ae6a79555da41a0f7bd02a04f0857275f8ee9 WatchSource:0}: Error finding container bbc52bdadca886c13d91b9f1a26ae6a79555da41a0f7bd02a04f0857275f8ee9: Status 404 returned error can't find the container with id bbc52bdadca886c13d91b9f1a26ae6a79555da41a0f7bd02a04f0857275f8ee9 Aug 02 09:34:04.639533 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.639498 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5"] Aug 02 09:34:04.641786 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:34:04.641763 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efa155f_3829_42ea_8166_e046d9e61392.slice/crio-66a20932648846b3c1b88566113bffcb9f6e5b275153e80992bc03d9c4563f08 WatchSource:0}: Error finding container 66a20932648846b3c1b88566113bffcb9f6e5b275153e80992bc03d9c4563f08: Status 404 returned error can't find the container with id 66a20932648846b3c1b88566113bffcb9f6e5b275153e80992bc03d9c4563f08 Aug 02 09:34:04.761819 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.761797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:34:04.761921 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.761840 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:34:04.763841 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.763791 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 02 09:34:04.763841 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.763810 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 02 09:34:04.774872 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.774767 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8dcf84-140a-4955-933b-6f40794f9b9e-metrics-certs\") pod \"network-metrics-daemon-wr257\" (UID: \"9f8dcf84-140a-4955-933b-6f40794f9b9e\") " pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:34:04.775219 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.775174 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27a5cbb8-9293-4a54-9016-a822e9633ed2-original-pull-secret\") pod \"global-pull-secret-syncer-qjr5v\" (UID: \"27a5cbb8-9293-4a54-9016-a822e9633ed2\") " pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:34:04.862935 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.862902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwv4\" (UniqueName: \"kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4\") pod \"network-check-target-68nsw\" (UID: \"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30\") " pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:34:04.863111 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.862951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.865146 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.865120 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 02 09:34:04.865380 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.865364 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/073c08eb-a4a8-490f-8af8-6453451cf4dd-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-fv4ph\" (UID: \"073c08eb-a4a8-490f-8af8-6453451cf4dd\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:04.875544 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.875529 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 02 09:34:04.886349 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.886329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvwv4\" (UniqueName: \"kubernetes.io/projected/75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30-kube-api-access-bvwv4\") pod \"network-check-target-68nsw\" (UID: \"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30\") " pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:34:04.984865 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.984844 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt8nz\"" Aug 02 09:34:04.988716 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.988703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjr5v" Aug 02 09:34:04.993246 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.993227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wr257" Aug 02 09:34:04.995033 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:04.995018 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mbl9n\"" Aug 02 09:34:05.004191 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.004171 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:34:05.140439 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.140355 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qjr5v"] Aug 02 09:34:05.142757 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.142515 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" Aug 02 09:34:05.161408 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.161380 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wr257"] Aug 02 09:34:05.167155 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:34:05.167123 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8dcf84_140a_4955_933b_6f40794f9b9e.slice/crio-cbaf36dc1ea616b6e00b495627e62bec3584995e0bdbd0d234b0e580c0a76637 WatchSource:0}: Error finding container cbaf36dc1ea616b6e00b495627e62bec3584995e0bdbd0d234b0e580c0a76637: Status 404 returned error can't find the container with id cbaf36dc1ea616b6e00b495627e62bec3584995e0bdbd0d234b0e580c0a76637 Aug 02 09:34:05.183864 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.183832 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-68nsw"] Aug 02 09:34:05.187504 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:34:05.187443 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d8cc48_54ca_4f7c_9066_3fb3c5c5ae30.slice/crio-b9a0598eb6a321afc0394a94e479976c329f7c2fe936d37f80ac66cd69ac311c WatchSource:0}: Error finding container b9a0598eb6a321afc0394a94e479976c329f7c2fe936d37f80ac66cd69ac311c: Status 404 returned error can't find the container with id b9a0598eb6a321afc0394a94e479976c329f7c2fe936d37f80ac66cd69ac311c Aug 02 09:34:05.219565 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.219538 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mdqdc" Aug 02 09:34:05.260028 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.259995 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-68nsw" event={"ID":"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30","Type":"ContainerStarted","Data":"b9a0598eb6a321afc0394a94e479976c329f7c2fe936d37f80ac66cd69ac311c"} Aug 02 09:34:05.269564 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.269521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" event={"ID":"2efa155f-3829-42ea-8166-e046d9e61392","Type":"ContainerStarted","Data":"f6b830929bf32ad368770b1df9470a38f5eaffbe9372e839dd7c07024f5a2c96"} Aug 02 09:34:05.269697 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.269608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" event={"ID":"2efa155f-3829-42ea-8166-e046d9e61392","Type":"ContainerStarted","Data":"0ce6f34198130c773f9a3870eb6a57657c1a039ec0acbbc61fc8dba0389575b1"} Aug 02 09:34:05.269697 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.269647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" event={"ID":"2efa155f-3829-42ea-8166-e046d9e61392","Type":"ContainerStarted","Data":"66a20932648846b3c1b88566113bffcb9f6e5b275153e80992bc03d9c4563f08"} Aug 02 09:34:05.271906 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.271866 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wr257" event={"ID":"9f8dcf84-140a-4955-933b-6f40794f9b9e","Type":"ContainerStarted","Data":"cbaf36dc1ea616b6e00b495627e62bec3584995e0bdbd0d234b0e580c0a76637"} Aug 02 09:34:05.275065 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.275040 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qjr5v" event={"ID":"27a5cbb8-9293-4a54-9016-a822e9633ed2","Type":"ContainerStarted","Data":"a8a16d338d5812712a6491701a19cf8413e71ccfd3ab98111eea356e96f0651e"} Aug 02 09:34:05.276899 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.276847 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kpkq" event={"ID":"47487674-5e42-4214-a3e9-59ebf2d12c7b","Type":"ContainerStarted","Data":"bbc52bdadca886c13d91b9f1a26ae6a79555da41a0f7bd02a04f0857275f8ee9"} Aug 02 09:34:05.287428 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.287402 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 02 09:34:05.292899 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.292844 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.296314 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.296139 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 02 09:34:05.296430 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.296326 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 02 09:34:05.296516 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.296490 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 02 09:34:05.296892 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.296870 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q6ln6\"" Aug 02 09:34:05.296990 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.296925 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 02 09:34:05.297158 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.297134 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 02 09:34:05.297444 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.297424 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 02 09:34:05.297549 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.297529 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 02 09:34:05.297770 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.297610 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 02 09:34:05.297770 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.296874 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 02 09:34:05.307987 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.307878 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph"] Aug 02 09:34:05.309383 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.309362 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 02 09:34:05.311695 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:34:05.311494 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073c08eb_a4a8_490f_8af8_6453451cf4dd.slice/crio-61b06bbacf4f8c6700bbaf61f1621b312a738f731683292b7015c622c59b9d8b WatchSource:0}: Error finding container 61b06bbacf4f8c6700bbaf61f1621b312a738f731683292b7015c622c59b9d8b: Status 404 returned error can't find the container with id 61b06bbacf4f8c6700bbaf61f1621b312a738f731683292b7015c622c59b9d8b Aug 02 09:34:05.368964 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.368931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369116 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.368978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369116 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.369019 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369116 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.369054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-web-config\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369116 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.369085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369266 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.369147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-out\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369266 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.369186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369266 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.369213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369397 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.369269 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krzk5\" (UniqueName: \"kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-kube-api-access-krzk5\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369397 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.369293 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369397 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.369345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-volume\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369397 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.369379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.369592 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.369434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.470898 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.470861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471066 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.470909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-web-config\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471218 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.471196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471278 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.471250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-out\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471327 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.471281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471327 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.471321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471420 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.471365 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krzk5\" (UniqueName: \"kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-kube-api-access-krzk5\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471420 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.471388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471420 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.471408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-volume\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471576 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.471429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471576 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.471495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471576 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.471527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.471576 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.471559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.472852 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:34:05.472234 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-trusted-ca-bundle podName:4cb66dae-05f3-4a52-8fcc-afd82ac33207 nodeName:}" failed. No retries permitted until 2026-08-02 09:34:05.972201645 +0000 UTC m=+67.413092100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207") : configmap references non-existent config key: ca-bundle.crt Aug 02 09:34:05.474784 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.473987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.474784 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.474724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.477232 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.477202 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.478231 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.478203 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-volume\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.478540 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.478516 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.479043 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.479008 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.479158 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.479078 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-web-config\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.479848 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.479825 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.480085 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.480017 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.480192 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.480166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-out\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.480893 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.480873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.483184 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.483150 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krzk5\" (UniqueName: \"kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-kube-api-access-krzk5\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.976896 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.976861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:05.977726 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:05.977701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:06.201939 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.201904 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5b59bc865b-6r822"] Aug 02 09:34:06.208268 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.208239 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:34:06.222756 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.222718 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b59bc865b-6r822"] Aug 02 09:34:06.223160 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.222773 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.225168 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.225110 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 02 09:34:06.225372 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.225344 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s84cs\"" Aug 02 09:34:06.225456 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.225389 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 02 09:34:06.225456 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.225344 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-74mfklh2hsf3o\"" Aug 02 09:34:06.225753 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.225732 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 02 09:34:06.225868 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.225798 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 02 09:34:06.225868 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.225833 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 02 09:34:06.279719 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.279682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.279719 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.279711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/315e54db-6885-4ae2-83a2-bed703db3b9d-metrics-client-ca\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.279940 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.279737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.279940 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.279764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-tls\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.279940 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.279780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-grpc-tls\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.279940 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.279842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25l5v\" (UniqueName: \"kubernetes.io/projected/315e54db-6885-4ae2-83a2-bed703db3b9d-kube-api-access-25l5v\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.279940 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.279890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.280159 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.279924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.282215 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.282183 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" event={"ID":"073c08eb-a4a8-490f-8af8-6453451cf4dd","Type":"ContainerStarted","Data":"61b06bbacf4f8c6700bbaf61f1621b312a738f731683292b7015c622c59b9d8b"} Aug 02 09:34:06.381437 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.381388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l5v\" (UniqueName: \"kubernetes.io/projected/315e54db-6885-4ae2-83a2-bed703db3b9d-kube-api-access-25l5v\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.381675 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.381481 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.381675 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.381577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.381675 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.381605 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.381675 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.381629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/315e54db-6885-4ae2-83a2-bed703db3b9d-metrics-client-ca\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.381675 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.381657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.382167 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.382055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-tls\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.382167 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.382095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-grpc-tls\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.382618 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.382532 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/315e54db-6885-4ae2-83a2-bed703db3b9d-metrics-client-ca\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.384648 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.384620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.387207 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.385208 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-tls\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.387207 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.385792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.387207 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.386002 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.388092 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.388050 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-grpc-tls\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.388756 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.388734 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/315e54db-6885-4ae2-83a2-bed703db3b9d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.389400 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.389361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25l5v\" (UniqueName: \"kubernetes.io/projected/315e54db-6885-4ae2-83a2-bed703db3b9d-kube-api-access-25l5v\") pod \"thanos-querier-5b59bc865b-6r822\" (UID: \"315e54db-6885-4ae2-83a2-bed703db3b9d\") " pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:06.535359 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:06.535280 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:07.065657 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:07.065547 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b59bc865b-6r822"] Aug 02 09:34:07.081951 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:07.081833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 02 09:34:07.291112 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:07.290189 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" event={"ID":"2efa155f-3829-42ea-8166-e046d9e61392","Type":"ContainerStarted","Data":"06f9fc0fe81fc8d8c0c96d1e11a375a0e52a98b73d9c1f0cf4198d005dac2061"} Aug 02 09:34:07.293218 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:07.293102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" event={"ID":"315e54db-6885-4ae2-83a2-bed703db3b9d","Type":"ContainerStarted","Data":"a7bc340a1e6ed88055aa17eb527f582bffa51307d8d618835d809eca512e0b3c"} Aug 02 09:34:07.295070 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:07.295015 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wr257" event={"ID":"9f8dcf84-140a-4955-933b-6f40794f9b9e","Type":"ContainerStarted","Data":"4712575a7427d931ab13bced210d9f62248ee4376d68e4e7d306bcdf8ea7b234"} Aug 02 09:34:07.299676 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:07.299492 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kpkq" event={"ID":"47487674-5e42-4214-a3e9-59ebf2d12c7b","Type":"ContainerStarted","Data":"bda11f8efdea569019ac168a3b32ed44b5ccae2ef7c6a74fb4bbec018c4a5c60"} Aug 02 09:34:07.309090 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:07.309029 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-dzcm5" podStartSLOduration=1.207794825 podStartE2EDuration="3.309012922s" totalImagesPullingTime="2.101218097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:07.308737065 +0000 UTC m=+68.749627524" watchObservedRunningTime="2026-08-02 09:34:07.309012922 +0000 UTC m=+68.749903383" Aug 02 09:34:08.304086 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.304052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wr257" event={"ID":"9f8dcf84-140a-4955-933b-6f40794f9b9e","Type":"ContainerStarted","Data":"3a9578f019c94dc6e2cb59aa380fb60b068fbfbe2337ee1f156e50ae8c85b37d"} Aug 02 09:34:08.306888 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.306859 2564 generic.go:356] "Generic (PLEG): container finished" podID="47487674-5e42-4214-a3e9-59ebf2d12c7b" containerID="bda11f8efdea569019ac168a3b32ed44b5ccae2ef7c6a74fb4bbec018c4a5c60" exitCode=0 Aug 02 09:34:08.307066 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.306993 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kpkq" event={"ID":"47487674-5e42-4214-a3e9-59ebf2d12c7b","Type":"ContainerDied","Data":"bda11f8efdea569019ac168a3b32ed44b5ccae2ef7c6a74fb4bbec018c4a5c60"} Aug 02 09:34:08.310295 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.309867 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerStarted","Data":"148db9536110d5ce02a980bbb1cd6749a0ec53a9c794900396030b5985c443fc"} Aug 02 09:34:08.316124 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.315830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" event={"ID":"073c08eb-a4a8-490f-8af8-6453451cf4dd","Type":"ContainerStarted","Data":"1554fee06e328c7119985848e4ec4ae53329244cd2a57ee17bc10090c5fa982e"} Aug 02 09:34:08.316124 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.315878 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" event={"ID":"073c08eb-a4a8-490f-8af8-6453451cf4dd","Type":"ContainerStarted","Data":"0d7d364163cc632d572d5c3511cc3a70f3a6fe227e2c5c3ccbf0c1c423bc6f8e"} Aug 02 09:34:08.316124 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.315906 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" event={"ID":"073c08eb-a4a8-490f-8af8-6453451cf4dd","Type":"ContainerStarted","Data":"51a690da8a68a3edf6b9041986896beddea957cc798cc7466a3bcd34cdccaf61"} Aug 02 09:34:08.320886 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.320603 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wr257" podStartSLOduration=67.612382941 podStartE2EDuration="1m9.32058417s" totalImagesPullingTime="1.708201229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:08.31925948 +0000 UTC m=+69.760149938" watchObservedRunningTime="2026-08-02 09:34:08.32058417 +0000 UTC m=+69.761474630" Aug 02 09:34:08.351499 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.350651 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-fv4ph" podStartSLOduration=2.053162543 podStartE2EDuration="4.350629665s" totalImagesPullingTime="2.297467122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:08.34969874 +0000 UTC m=+69.790589209" watchObservedRunningTime="2026-08-02 09:34:08.350629665 +0000 UTC m=+69.791520122" Aug 02 09:34:08.522520 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.522483 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f75c78d86-6wzwh"] Aug 02 09:34:08.526622 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.525783 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.528379 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.528356 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3jc909t9s7l87\"" Aug 02 09:34:08.528721 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.528701 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ch4ls\"" Aug 02 09:34:08.529599 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.529160 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 02 09:34:08.529599 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.529456 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Aug 02 09:34:08.529754 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.529674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Aug 02 09:34:08.529876 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.529854 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Aug 02 09:34:08.532733 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.532711 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f75c78d86-6wzwh"] Aug 02 09:34:08.603674 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.603591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c2d61a54-8add-43e7-bc0d-f6752f8444f7-client-ca-bundle\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.603674 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.603652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c2d61a54-8add-43e7-bc0d-f6752f8444f7-secret-metrics-server-client-certs\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.603881 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.603683 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c2d61a54-8add-43e7-bc0d-f6752f8444f7-metrics-server-audit-profiles\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.603881 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.603719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d61a54-8add-43e7-bc0d-f6752f8444f7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.603881 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.603783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c2d61a54-8add-43e7-bc0d-f6752f8444f7-secret-metrics-server-tls\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.603881 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.603837 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48twl\" (UniqueName: \"kubernetes.io/projected/c2d61a54-8add-43e7-bc0d-f6752f8444f7-kube-api-access-48twl\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.603881 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.603862 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c2d61a54-8add-43e7-bc0d-f6752f8444f7-audit-log\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.705016 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.704780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c2d61a54-8add-43e7-bc0d-f6752f8444f7-secret-metrics-server-client-certs\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.705016 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.704833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c2d61a54-8add-43e7-bc0d-f6752f8444f7-metrics-server-audit-profiles\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.705016 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.704870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d61a54-8add-43e7-bc0d-f6752f8444f7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.705016 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.704933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c2d61a54-8add-43e7-bc0d-f6752f8444f7-secret-metrics-server-tls\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.705016 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.704993 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48twl\" (UniqueName: \"kubernetes.io/projected/c2d61a54-8add-43e7-bc0d-f6752f8444f7-kube-api-access-48twl\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.705016 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.705018 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c2d61a54-8add-43e7-bc0d-f6752f8444f7-audit-log\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.705447 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.705050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c2d61a54-8add-43e7-bc0d-f6752f8444f7-client-ca-bundle\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.706007 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.705670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2d61a54-8add-43e7-bc0d-f6752f8444f7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.706007 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.705932 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c2d61a54-8add-43e7-bc0d-f6752f8444f7-metrics-server-audit-profiles\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.706007 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.705973 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c2d61a54-8add-43e7-bc0d-f6752f8444f7-audit-log\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.707995 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.707951 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c2d61a54-8add-43e7-bc0d-f6752f8444f7-client-ca-bundle\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.708458 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.708414 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c2d61a54-8add-43e7-bc0d-f6752f8444f7-secret-metrics-server-client-certs\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.708655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.708632 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c2d61a54-8add-43e7-bc0d-f6752f8444f7-secret-metrics-server-tls\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.712410 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.712366 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48twl\" (UniqueName: \"kubernetes.io/projected/c2d61a54-8add-43e7-bc0d-f6752f8444f7-kube-api-access-48twl\") pod \"metrics-server-f75c78d86-6wzwh\" (UID: \"c2d61a54-8add-43e7-bc0d-f6752f8444f7\") " pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.842958 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.842920 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:08.993390 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.993356 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf"] Aug 02 09:34:08.999855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:08.999827 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" Aug 02 09:34:09.002618 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.002331 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 02 09:34:09.002741 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.002672 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5lk8v\"" Aug 02 09:34:09.003492 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.003454 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf"] Aug 02 09:34:09.116814 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.116777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5zvz\" (UniqueName: \"kubernetes.io/projected/ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7-kube-api-access-q5zvz\") pod \"monitoring-plugin-6dffbbd67-w95wf\" (UID: \"ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" Aug 02 09:34:09.116985 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.116839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-w95wf\" (UID: \"ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" Aug 02 09:34:09.218028 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.217993 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5zvz\" (UniqueName: \"kubernetes.io/projected/ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7-kube-api-access-q5zvz\") pod \"monitoring-plugin-6dffbbd67-w95wf\" (UID: \"ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" Aug 02 09:34:09.218215 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.218055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-w95wf\" (UID: \"ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" Aug 02 09:34:09.218215 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:34:09.218197 2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Aug 02 09:34:09.218327 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:34:09.218258 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7-monitoring-plugin-cert podName:ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7 nodeName:}" failed. No retries permitted until 2026-08-02 09:34:09.718236523 +0000 UTC m=+71.159126956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7-monitoring-plugin-cert") pod "monitoring-plugin-6dffbbd67-w95wf" (UID: "ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7") : secret "monitoring-plugin-cert" not found Aug 02 09:34:09.228410 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.228379 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5zvz\" (UniqueName: \"kubernetes.io/projected/ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7-kube-api-access-q5zvz\") pod \"monitoring-plugin-6dffbbd67-w95wf\" (UID: \"ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" Aug 02 09:34:09.323113 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.323030 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kpkq" event={"ID":"47487674-5e42-4214-a3e9-59ebf2d12c7b","Type":"ContainerStarted","Data":"3822c5e90a1b9c95b9aa5a2615a02f7b71328c2e1c0d6c3c880a69bfe88d09b6"} Aug 02 09:34:09.323520 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.323138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5kpkq" event={"ID":"47487674-5e42-4214-a3e9-59ebf2d12c7b","Type":"ContainerStarted","Data":"57bf5f0e939172f16ddd2bfe064d068df5271ed0b41953d1bc7af95aeb07d929"} Aug 02 09:34:09.339322 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.339271 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5kpkq" podStartSLOduration=2.952179662 podStartE2EDuration="5.33925476s" totalImagesPullingTime="2.300251157s" totalInitContainerRuntime="86.823941ms" isStatefulPod=true podCreationTimestamp="2026-08-02 09:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:09.338624035 +0000 UTC m=+70.779514492" watchObservedRunningTime="2026-08-02 09:34:09.33925476 +0000 UTC m=+70.780145216" Aug 02 09:34:09.723558 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.723515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-w95wf\" (UID: \"ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" Aug 02 09:34:09.726196 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.726172 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-w95wf\" (UID: \"ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" Aug 02 09:34:09.914822 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:09.914792 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" Aug 02 09:34:10.392323 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.392287 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 02 09:34:10.398534 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.396944 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.399888 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.399859 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.400089 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.400346 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lqcrs\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.400767 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.400769 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.400965 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.401180 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.401366 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.401425 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.401526 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.401627 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.401716 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a9uus8plrndk3\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.402083 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.402225 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 02 09:34:10.402385 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.402279 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 02 09:34:10.403994 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.403970 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 02 09:34:10.410278 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.410257 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 02 09:34:10.530440 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.530440 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530438 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.530712 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.530712 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.530712 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.530712 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-config\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.530712 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530626 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-config-out\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.530712 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grrc\" (UniqueName: \"kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-kube-api-access-4grrc\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.530712 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.531063 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.531063 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-web-config\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.531063 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.531063 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530854 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.531063 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530881 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.531063 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.531063 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.531063 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.530980 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.531063 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.531005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.531063 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.531054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.531593 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.531082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632339 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632530 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632530 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-web-config\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632530 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632530 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632530 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632788 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632788 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632788 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632788 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632788 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632788 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632788 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632729 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632788 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.632788 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.633159 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.633159 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.633159 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-config\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.633159 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632888 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-config-out\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.633159 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.632914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grrc\" (UniqueName: \"kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-kube-api-access-4grrc\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.633509 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.633452 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.633895 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.633867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.634069 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.634044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.634570 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.634549 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.635144 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.635118 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.635553 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.635529 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.635648 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.635562 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.636223 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.636199 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.636732 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.636703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.637262 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.637232 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.638162 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.637803 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-config-out\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.638162 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.638121 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-web-config\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.638522 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.638316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.640221 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.640079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-config\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.642605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.641841 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grrc\" (UniqueName: \"kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-kube-api-access-4grrc\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.642605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.641976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.642605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.642430 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.642831 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.642808 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.643102 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.643082 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.643171 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.643110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:10.713829 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:10.713790 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:12.220236 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.219625 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f75c78d86-6wzwh"] Aug 02 09:34:12.289475 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:34:12.289427 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d61a54_8add_43e7_bc0d_f6752f8444f7.slice/crio-b19bf952427c9fa479acd0435fa140065c5915b51c4d5e78790ffc28ea759497 WatchSource:0}: Error finding container b19bf952427c9fa479acd0435fa140065c5915b51c4d5e78790ffc28ea759497: Status 404 returned error can't find the container with id b19bf952427c9fa479acd0435fa140065c5915b51c4d5e78790ffc28ea759497 Aug 02 09:34:12.338163 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.338129 2564 generic.go:356] "Generic (PLEG): container finished" podID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerID="1b46382128f6259b6eba6e4961101319f721d54875ca685a633093cbcc13e409" exitCode=0 Aug 02 09:34:12.338263 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.338222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerDied","Data":"1b46382128f6259b6eba6e4961101319f721d54875ca685a633093cbcc13e409"} Aug 02 09:34:12.339730 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.339699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" event={"ID":"c2d61a54-8add-43e7-bc0d-f6752f8444f7","Type":"ContainerStarted","Data":"b19bf952427c9fa479acd0435fa140065c5915b51c4d5e78790ffc28ea759497"} Aug 02 09:34:12.341882 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.341853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-68nsw" event={"ID":"75d8cc48-54ca-4f7c-9066-3fb3c5c5ae30","Type":"ContainerStarted","Data":"f949124978b5b120faa189765ae91f05fe7f4fa630ef61a083fa924f1e9e03d6"} Aug 02 09:34:12.341990 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.341927 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:34:12.344836 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.344747 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" event={"ID":"315e54db-6885-4ae2-83a2-bed703db3b9d","Type":"ContainerStarted","Data":"241d1455585364c0a78c02342754f453101ff1d1a992b1c7ee6f053bafc784a4"} Aug 02 09:34:12.344836 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.344802 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" event={"ID":"315e54db-6885-4ae2-83a2-bed703db3b9d","Type":"ContainerStarted","Data":"723cf9c5a8783dd45682e82b40cf066427b9e0b6c112528ce348b7549b8cdc7c"} Aug 02 09:34:12.346584 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.346561 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qjr5v" event={"ID":"27a5cbb8-9293-4a54-9016-a822e9633ed2","Type":"ContainerStarted","Data":"efff9d89f30c5b4189d025789a2f6ab8150b0669f1dfec619fd0ab64e4058d36"} Aug 02 09:34:12.374983 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.374942 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-68nsw" podStartSLOduration=66.525005992 podStartE2EDuration="1m13.374925874s" totalImagesPullingTime="6.849919882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:12.373846041 +0000 UTC m=+73.814736498" watchObservedRunningTime="2026-08-02 09:34:12.374925874 +0000 UTC m=+73.815816603" Aug 02 09:34:12.387697 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.387658 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qjr5v" podStartSLOduration=66.487529162 podStartE2EDuration="1m13.387644215s" totalImagesPullingTime="6.900115053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 09:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:12.386574614 +0000 UTC m=+73.827465071" watchObservedRunningTime="2026-08-02 09:34:12.387644215 +0000 UTC m=+73.828534670" Aug 02 09:34:12.440140 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.440100 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf"] Aug 02 09:34:12.442244 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:34:12.442207 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdee2fd_65c3_4b43_abe8_ddd5b7b46af7.slice/crio-a72f656da726274a34f30343165d39e4a55e6f57ff6703f91558fc760ca54c28 WatchSource:0}: Error finding container a72f656da726274a34f30343165d39e4a55e6f57ff6703f91558fc760ca54c28: Status 404 returned error can't find the container with id a72f656da726274a34f30343165d39e4a55e6f57ff6703f91558fc760ca54c28 Aug 02 09:34:12.455561 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:12.455538 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 02 09:34:12.456444 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:34:12.456424 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf036671c_4718_4c1c_8384_7c350be8a8fe.slice/crio-773a27f19e30ce091351310f57e1fb37cd2df9d3672e64d7e41aeb13c4b6269d WatchSource:0}: Error finding container 773a27f19e30ce091351310f57e1fb37cd2df9d3672e64d7e41aeb13c4b6269d: Status 404 returned error can't find the container with id 773a27f19e30ce091351310f57e1fb37cd2df9d3672e64d7e41aeb13c4b6269d Aug 02 09:34:13.354905 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:13.354818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" event={"ID":"ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7","Type":"ContainerStarted","Data":"a72f656da726274a34f30343165d39e4a55e6f57ff6703f91558fc760ca54c28"} Aug 02 09:34:13.358158 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:13.357779 2564 generic.go:356] "Generic (PLEG): container finished" podID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerID="a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4" exitCode=0 Aug 02 09:34:13.358158 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:13.357853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerDied","Data":"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4"} Aug 02 09:34:13.358158 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:13.357925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerStarted","Data":"773a27f19e30ce091351310f57e1fb37cd2df9d3672e64d7e41aeb13c4b6269d"} Aug 02 09:34:13.363365 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:13.362879 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" event={"ID":"315e54db-6885-4ae2-83a2-bed703db3b9d","Type":"ContainerStarted","Data":"83559e69247c89d2d82b1acc2eacfa00260763776a0ffdb7574666099bfde45c"} Aug 02 09:34:14.209737 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:14.209704 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c7cbb5c7f-q759r" Aug 02 09:34:14.368951 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:14.368863 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" event={"ID":"315e54db-6885-4ae2-83a2-bed703db3b9d","Type":"ContainerStarted","Data":"1ef3749f5eb68bbb51e14a30a11600fd633b3bc4ebd72c308fdb97bc91dc57a8"} Aug 02 09:34:15.378718 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.378671 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerStarted","Data":"2a1bfa6049b7aaa7ac8d35422798bd47f0390c79e723b071d0600c78a9507fb7"} Aug 02 09:34:15.379161 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.378761 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerStarted","Data":"98c0fb7c9efa6342878355e66f8d282791d6d1c073b3627e5162d905ddf2e54d"} Aug 02 09:34:15.379161 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.378803 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerStarted","Data":"78790aabf22e2782160fb520874615379dea4bb169e5602930a0b8385e46386a"} Aug 02 09:34:15.379161 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.378842 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerStarted","Data":"8cf3c6baad3f732e6dbc8dd5b690771e7064499da796892e9aa30530383bf1a6"} Aug 02 09:34:15.379161 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.378888 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerStarted","Data":"f2d43ed2748a2471262c954e9488a190eaeda3f8760e10c198b2ee0b5e75b9bc"} Aug 02 09:34:15.379161 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.378924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerStarted","Data":"0b91e97f85790352c17f5fba89bfcbcd7746291cec61319b2878f9442dc2c567"} Aug 02 09:34:15.380595 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.380520 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" event={"ID":"c2d61a54-8add-43e7-bc0d-f6752f8444f7","Type":"ContainerStarted","Data":"c33efd0151f97282b2cb097f42be9f66a2622f550edb2325ef9f8b6551f66c5f"} Aug 02 09:34:15.382298 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.382270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" event={"ID":"ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7","Type":"ContainerStarted","Data":"81c60b25a34dd82dc0c4a33d029044353c55275a346e0327aeec4fe85e5ea34a"} Aug 02 09:34:15.382489 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.382449 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" Aug 02 09:34:15.386191 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.386130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" event={"ID":"315e54db-6885-4ae2-83a2-bed703db3b9d","Type":"ContainerStarted","Data":"087e678e152722cf80a430a1be3b481f62d5d7f806e022dbb96173e71109b242"} Aug 02 09:34:15.386284 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.386194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" event={"ID":"315e54db-6885-4ae2-83a2-bed703db3b9d","Type":"ContainerStarted","Data":"b007e0ac2fa92eea0c56d1dbd9d9963c3a404cb997945f4f43386e1c3a626eae"} Aug 02 09:34:15.386343 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.386328 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:15.389046 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.389023 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" Aug 02 09:34:15.403933 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.403885 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.734219222 podStartE2EDuration="10.403867506s" totalImagesPullingTime="6.644851239s" totalInitContainerRuntime="24.797045ms" isStatefulPod=false podCreationTimestamp="2026-08-02 09:34:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:15.402489679 +0000 UTC m=+76.843380132" watchObservedRunningTime="2026-08-02 09:34:15.403867506 +0000 UTC m=+76.844757963" Aug 02 09:34:15.439617 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.439567 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" podStartSLOduration=3.477769372 podStartE2EDuration="9.439548361s" totalImagesPullingTime="5.961778989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:34:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:15.424307414 +0000 UTC m=+76.865197872" watchObservedRunningTime="2026-08-02 09:34:15.439548361 +0000 UTC m=+76.880438816" Aug 02 09:34:15.441082 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.441024 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" podStartSLOduration=5.206424259 podStartE2EDuration="7.441011356s" totalImagesPullingTime="2.234587097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:15.438391312 +0000 UTC m=+76.879281769" watchObservedRunningTime="2026-08-02 09:34:15.441011356 +0000 UTC m=+76.881901813" Aug 02 09:34:15.453451 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:15.453402 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-w95wf" podStartSLOduration=5.371076415 podStartE2EDuration="7.453384172s" totalImagesPullingTime="2.082307757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:15.452805681 +0000 UTC m=+76.893696138" watchObservedRunningTime="2026-08-02 09:34:15.453384172 +0000 UTC m=+76.894274631" Aug 02 09:34:17.399067 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:17.399001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerStarted","Data":"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe"} Aug 02 09:34:17.399362 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:17.399156 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerStarted","Data":"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c"} Aug 02 09:34:17.399362 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:17.399188 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerStarted","Data":"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005"} Aug 02 09:34:18.405514 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:18.405475 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerStarted","Data":"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e"} Aug 02 09:34:18.405879 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:18.405556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerStarted","Data":"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7"} Aug 02 09:34:18.405879 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:18.405596 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerStarted","Data":"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257"} Aug 02 09:34:18.432388 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:18.432328 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.645821489 podStartE2EDuration="8.432306793s" totalImagesPullingTime="3.768138264s" totalInitContainerRuntime="18.34704ms" isStatefulPod=false podCreationTimestamp="2026-08-02 09:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:34:18.430144279 +0000 UTC m=+79.871034745" watchObservedRunningTime="2026-08-02 09:34:18.432306793 +0000 UTC m=+79.873197250" Aug 02 09:34:20.714363 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:20.714310 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:34:21.396689 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:21.396664 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b59bc865b-6r822" Aug 02 09:34:28.843804 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:28.843767 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:28.844208 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:28.843830 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:43.365694 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:43.365663 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-68nsw" Aug 02 09:34:48.848751 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:48.848715 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:48.852690 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:48.852665 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f75c78d86-6wzwh" Aug 02 09:34:50.375196 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:50.375164 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4cb66dae-05f3-4a52-8fcc-afd82ac33207/init-config-reloader/0.log" Aug 02 09:34:50.381089 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:50.381068 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4cb66dae-05f3-4a52-8fcc-afd82ac33207/alertmanager/0.log" Aug 02 09:34:50.541874 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:50.541846 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4cb66dae-05f3-4a52-8fcc-afd82ac33207/config-reloader/0.log" Aug 02 09:34:50.742317 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:50.742280 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4cb66dae-05f3-4a52-8fcc-afd82ac33207/kube-rbac-proxy-web/0.log" Aug 02 09:34:50.942275 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:50.942241 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4cb66dae-05f3-4a52-8fcc-afd82ac33207/kube-rbac-proxy/0.log" Aug 02 09:34:51.142745 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:51.142678 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4cb66dae-05f3-4a52-8fcc-afd82ac33207/kube-rbac-proxy-metric/0.log" Aug 02 09:34:51.342726 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:51.342689 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4cb66dae-05f3-4a52-8fcc-afd82ac33207/prom-label-proxy/0.log" Aug 02 09:34:51.743339 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:51.743308 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-fv4ph_073c08eb-a4a8-490f-8af8-6453451cf4dd/kube-state-metrics/0.log" Aug 02 09:34:51.942191 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:51.942153 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-fv4ph_073c08eb-a4a8-490f-8af8-6453451cf4dd/kube-rbac-proxy-main/0.log" Aug 02 09:34:52.142923 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:52.142852 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-fv4ph_073c08eb-a4a8-490f-8af8-6453451cf4dd/kube-rbac-proxy-self/0.log" Aug 02 09:34:52.343255 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:52.343224 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f75c78d86-6wzwh_c2d61a54-8add-43e7-bc0d-f6752f8444f7/metrics-server/0.log" Aug 02 09:34:52.544408 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:52.544384 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-w95wf_ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7/monitoring-plugin/0.log" Aug 02 09:34:52.741881 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:52.741859 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kpkq_47487674-5e42-4214-a3e9-59ebf2d12c7b/init-textfile/0.log" Aug 02 09:34:52.943787 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:52.943717 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kpkq_47487674-5e42-4214-a3e9-59ebf2d12c7b/node-exporter/0.log" Aug 02 09:34:53.142160 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:53.142135 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kpkq_47487674-5e42-4214-a3e9-59ebf2d12c7b/kube-rbac-proxy/0.log" Aug 02 09:34:54.542398 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:54.542371 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-dzcm5_2efa155f-3829-42ea-8166-e046d9e61392/kube-rbac-proxy-main/0.log" Aug 02 09:34:54.743992 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:54.743963 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-dzcm5_2efa155f-3829-42ea-8166-e046d9e61392/kube-rbac-proxy-self/0.log" Aug 02 09:34:54.942843 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:54.942772 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-dzcm5_2efa155f-3829-42ea-8166-e046d9e61392/openshift-state-metrics/0.log" Aug 02 09:34:55.143037 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:55.143004 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f036671c-4718-4c1c-8384-7c350be8a8fe/init-config-reloader/0.log" Aug 02 09:34:55.344694 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:55.344657 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f036671c-4718-4c1c-8384-7c350be8a8fe/prometheus/0.log" Aug 02 09:34:55.545920 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:55.545892 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f036671c-4718-4c1c-8384-7c350be8a8fe/config-reloader/0.log" Aug 02 09:34:55.742729 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:55.742705 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f036671c-4718-4c1c-8384-7c350be8a8fe/thanos-sidecar/0.log" Aug 02 09:34:55.942515 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:55.942491 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f036671c-4718-4c1c-8384-7c350be8a8fe/kube-rbac-proxy-web/0.log" Aug 02 09:34:56.142871 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:56.142804 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f036671c-4718-4c1c-8384-7c350be8a8fe/kube-rbac-proxy/0.log" Aug 02 09:34:56.342843 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:56.342814 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f036671c-4718-4c1c-8384-7c350be8a8fe/kube-rbac-proxy-thanos/0.log" Aug 02 09:34:56.543821 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:56.543795 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-8fvdr_a4ff2e2c-ba72-49e2-b161-e11788ee3560/prometheus-operator/0.log" Aug 02 09:34:56.742820 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:56.742792 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-8fvdr_a4ff2e2c-ba72-49e2-b161-e11788ee3560/kube-rbac-proxy/0.log" Aug 02 09:34:56.942606 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:56.942531 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-ffr77_abe2537e-513b-4ae4-9ae0-8632d800488a/prometheus-operator-admission-webhook/0.log" Aug 02 09:34:57.142593 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:57.142564 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/thanos-query/0.log" Aug 02 09:34:57.342752 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:57.342726 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/kube-rbac-proxy-web/0.log" Aug 02 09:34:57.542759 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:57.542733 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/kube-rbac-proxy/0.log" Aug 02 09:34:57.742926 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:57.742897 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/prom-label-proxy/0.log" Aug 02 09:34:57.942134 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:57.942103 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/kube-rbac-proxy-rules/0.log" Aug 02 09:34:58.142358 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:34:58.142296 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/kube-rbac-proxy-metrics/0.log" Aug 02 09:35:10.714626 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:10.714590 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:10.733037 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:10.733011 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:11.642104 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:11.642082 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:24.475104 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.475062 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 02 09:35:24.475590 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.475447 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="alertmanager" containerID="cri-o://0b91e97f85790352c17f5fba89bfcbcd7746291cec61319b2878f9442dc2c567" gracePeriod=120 Aug 02 09:35:24.475590 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.475513 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy-web" containerID="cri-o://8cf3c6baad3f732e6dbc8dd5b690771e7064499da796892e9aa30530383bf1a6" gracePeriod=120 Aug 02 09:35:24.475590 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.475532 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="config-reloader" containerID="cri-o://f2d43ed2748a2471262c954e9488a190eaeda3f8760e10c198b2ee0b5e75b9bc" gracePeriod=120 Aug 02 09:35:24.475798 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.475532 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="prom-label-proxy" containerID="cri-o://2a1bfa6049b7aaa7ac8d35422798bd47f0390c79e723b071d0600c78a9507fb7" gracePeriod=120 Aug 02 09:35:24.475798 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.475564 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy" containerID="cri-o://78790aabf22e2782160fb520874615379dea4bb169e5602930a0b8385e46386a" gracePeriod=120 Aug 02 09:35:24.475798 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.475477 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy-metric" containerID="cri-o://98c0fb7c9efa6342878355e66f8d282791d6d1c073b3627e5162d905ddf2e54d" gracePeriod=120 Aug 02 09:35:24.679606 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.679578 2564 generic.go:356] "Generic (PLEG): container finished" podID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerID="2a1bfa6049b7aaa7ac8d35422798bd47f0390c79e723b071d0600c78a9507fb7" exitCode=0 Aug 02 09:35:24.679606 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.679603 2564 generic.go:356] "Generic (PLEG): container finished" podID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerID="78790aabf22e2782160fb520874615379dea4bb169e5602930a0b8385e46386a" exitCode=0 Aug 02 09:35:24.679606 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.679609 2564 generic.go:356] "Generic (PLEG): container finished" podID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerID="f2d43ed2748a2471262c954e9488a190eaeda3f8760e10c198b2ee0b5e75b9bc" exitCode=0 Aug 02 09:35:24.679779 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.679615 2564 generic.go:356] "Generic (PLEG): container finished" podID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerID="0b91e97f85790352c17f5fba89bfcbcd7746291cec61319b2878f9442dc2c567" exitCode=0 Aug 02 09:35:24.679779 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.679661 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerDied","Data":"2a1bfa6049b7aaa7ac8d35422798bd47f0390c79e723b071d0600c78a9507fb7"} Aug 02 09:35:24.679779 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.679741 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerDied","Data":"78790aabf22e2782160fb520874615379dea4bb169e5602930a0b8385e46386a"} Aug 02 09:35:24.679779 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.679766 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerDied","Data":"f2d43ed2748a2471262c954e9488a190eaeda3f8760e10c198b2ee0b5e75b9bc"} Aug 02 09:35:24.679900 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:24.679790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerDied","Data":"0b91e97f85790352c17f5fba89bfcbcd7746291cec61319b2878f9442dc2c567"} Aug 02 09:35:25.687560 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.687527 2564 generic.go:356] "Generic (PLEG): container finished" podID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerID="98c0fb7c9efa6342878355e66f8d282791d6d1c073b3627e5162d905ddf2e54d" exitCode=0 Aug 02 09:35:25.687560 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.687552 2564 generic.go:356] "Generic (PLEG): container finished" podID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerID="8cf3c6baad3f732e6dbc8dd5b690771e7064499da796892e9aa30530383bf1a6" exitCode=0 Aug 02 09:35:25.687960 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.687596 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerDied","Data":"98c0fb7c9efa6342878355e66f8d282791d6d1c073b3627e5162d905ddf2e54d"} Aug 02 09:35:25.687960 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.687658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerDied","Data":"8cf3c6baad3f732e6dbc8dd5b690771e7064499da796892e9aa30530383bf1a6"} Aug 02 09:35:25.709852 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.709829 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:25.861984 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.861900 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-main-db\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.861984 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.861945 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.861984 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.861976 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-web-config\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.862245 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862000 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-out\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.862245 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862052 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-volume\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.862245 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862084 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krzk5\" (UniqueName: \"kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-kube-api-access-krzk5\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.862245 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862128 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-main-tls\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.862443 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862312 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:35:25.862526 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862456 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.862583 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862532 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-tls-assets\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.862638 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862584 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-trusted-ca-bundle\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.862638 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862611 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-web\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.862753 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862643 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-cluster-tls-config\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.862753 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862677 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-metrics-client-ca\") pod \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\" (UID: \"4cb66dae-05f3-4a52-8fcc-afd82ac33207\") " Aug 02 09:35:25.863010 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.862989 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-main-db\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.863371 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.863346 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 09:35:25.865301 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.865017 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:25.865301 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.865274 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:25.865542 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.865313 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-out" (OuterVolumeSpecName: "config-out") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:35:25.865542 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.865369 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:25.865659 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.865560 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 09:35:25.865718 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.865660 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 09:35:25.865843 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.865819 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-kube-api-access-krzk5" (OuterVolumeSpecName: "kube-api-access-krzk5") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "kube-api-access-krzk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 09:35:25.865943 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.865919 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-volume" (OuterVolumeSpecName: "config-volume") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:25.867572 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.867550 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:25.870828 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.870804 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:25.876810 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.876787 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-web-config" (OuterVolumeSpecName: "web-config") pod "4cb66dae-05f3-4a52-8fcc-afd82ac33207" (UID: "4cb66dae-05f3-4a52-8fcc-afd82ac33207"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:25.964229 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964200 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-web-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.964229 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964223 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-out\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.964229 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964232 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-config-volume\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.964406 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964240 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krzk5\" (UniqueName: \"kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-kube-api-access-krzk5\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.964406 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964250 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-main-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.964406 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964259 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.964406 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964268 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4cb66dae-05f3-4a52-8fcc-afd82ac33207-tls-assets\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.964406 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964278 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.964406 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964286 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.964406 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964296 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-cluster-tls-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.964406 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964304 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cb66dae-05f3-4a52-8fcc-afd82ac33207-metrics-client-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:25.964406 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:25.964313 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4cb66dae-05f3-4a52-8fcc-afd82ac33207-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:26.694758 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.694733 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.695123 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.694732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4cb66dae-05f3-4a52-8fcc-afd82ac33207","Type":"ContainerDied","Data":"148db9536110d5ce02a980bbb1cd6749a0ec53a9c794900396030b5985c443fc"} Aug 02 09:35:26.695123 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.694880 2564 scope.go:117] "RemoveContainer" containerID="2a1bfa6049b7aaa7ac8d35422798bd47f0390c79e723b071d0600c78a9507fb7" Aug 02 09:35:26.702340 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.702321 2564 scope.go:117] "RemoveContainer" containerID="98c0fb7c9efa6342878355e66f8d282791d6d1c073b3627e5162d905ddf2e54d" Aug 02 09:35:26.709620 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.709603 2564 scope.go:117] "RemoveContainer" containerID="78790aabf22e2782160fb520874615379dea4bb169e5602930a0b8385e46386a" Aug 02 09:35:26.715972 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.715957 2564 scope.go:117] "RemoveContainer" containerID="8cf3c6baad3f732e6dbc8dd5b690771e7064499da796892e9aa30530383bf1a6" Aug 02 09:35:26.720230 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.720206 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 02 09:35:26.722739 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.722716 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 02 09:35:26.724671 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.724657 2564 scope.go:117] "RemoveContainer" containerID="f2d43ed2748a2471262c954e9488a190eaeda3f8760e10c198b2ee0b5e75b9bc" Aug 02 09:35:26.731236 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.731220 2564 scope.go:117] "RemoveContainer" containerID="0b91e97f85790352c17f5fba89bfcbcd7746291cec61319b2878f9442dc2c567" Aug 02 09:35:26.737736 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.737715 2564 scope.go:117] "RemoveContainer" containerID="1b46382128f6259b6eba6e4961101319f721d54875ca685a633093cbcc13e409" Aug 02 09:35:26.746699 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.746680 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 02 09:35:26.746988 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.746975 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy-web" Aug 02 09:35:26.747030 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.746988 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy-web" Aug 02 09:35:26.747030 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.746997 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="config-reloader" Aug 02 09:35:26.747030 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747003 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="config-reloader" Aug 02 09:35:26.747030 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747010 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy" Aug 02 09:35:26.747030 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747015 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy" Aug 02 09:35:26.747030 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747021 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy-metric" Aug 02 09:35:26.747030 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747026 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy-metric" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747041 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="prom-label-proxy" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747046 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="prom-label-proxy" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747053 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="alertmanager" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747058 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="alertmanager" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747067 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="init-config-reloader" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747072 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="init-config-reloader" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747120 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="config-reloader" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747128 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747134 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="alertmanager" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747139 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="prom-label-proxy" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747146 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy-web" Aug 02 09:35:26.747227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.747153 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" containerName="kube-rbac-proxy-metric" Aug 02 09:35:26.749858 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.749842 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.752144 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.752123 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q6ln6\"" Aug 02 09:35:26.752253 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.752215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 02 09:35:26.752655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.752505 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 02 09:35:26.752655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.752532 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 02 09:35:26.752655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.752581 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 02 09:35:26.752655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.752641 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 02 09:35:26.752890 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.752795 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 02 09:35:26.752952 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.752896 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 02 09:35:26.753243 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.753221 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 02 09:35:26.757844 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.757828 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 02 09:35:26.767742 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.767719 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 02 09:35:26.871520 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/47233862-95a6-47dd-b064-4385f0aca30b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.871520 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871458 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47233862-95a6-47dd-b064-4385f0aca30b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.871725 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871546 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.871725 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871572 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.871725 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vl4\" (UniqueName: \"kubernetes.io/projected/47233862-95a6-47dd-b064-4385f0aca30b-kube-api-access-p6vl4\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.871725 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.871725 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871687 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-web-config\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.871725 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47233862-95a6-47dd-b064-4385f0aca30b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.872012 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-config-volume\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.872012 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47233862-95a6-47dd-b064-4385f0aca30b-config-out\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.872012 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.872012 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.872012 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.871953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47233862-95a6-47dd-b064-4385f0aca30b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.972903 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.972870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47233862-95a6-47dd-b064-4385f0aca30b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.972903 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.972903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-config-volume\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.973110 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.972922 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47233862-95a6-47dd-b064-4385f0aca30b-config-out\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.973110 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.972946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.973110 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.972981 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.973110 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.973002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47233862-95a6-47dd-b064-4385f0aca30b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.973110 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.973057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/47233862-95a6-47dd-b064-4385f0aca30b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.973110 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.973085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47233862-95a6-47dd-b064-4385f0aca30b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.973386 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.973114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.973386 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.973140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.973386 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.973173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vl4\" (UniqueName: \"kubernetes.io/projected/47233862-95a6-47dd-b064-4385f0aca30b-kube-api-access-p6vl4\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.973386 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.973205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.973386 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.973232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-web-config\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.976682 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.974687 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47233862-95a6-47dd-b064-4385f0aca30b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.976682 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.974709 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/47233862-95a6-47dd-b064-4385f0aca30b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.976682 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.976087 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.976682 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.976133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47233862-95a6-47dd-b064-4385f0aca30b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.976682 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.976199 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-config-volume\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.976682 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.976637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.977034 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.976699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.977034 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.976909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47233862-95a6-47dd-b064-4385f0aca30b-config-out\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.977342 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.977318 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47233862-95a6-47dd-b064-4385f0aca30b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.977651 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.977633 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-web-config\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.978084 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.978053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.978236 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.978218 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/47233862-95a6-47dd-b064-4385f0aca30b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:26.979889 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.979859 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cb66dae-05f3-4a52-8fcc-afd82ac33207" path="/var/lib/kubelet/pods/4cb66dae-05f3-4a52-8fcc-afd82ac33207/volumes" Aug 02 09:35:26.981210 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:26.981187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vl4\" (UniqueName: \"kubernetes.io/projected/47233862-95a6-47dd-b064-4385f0aca30b-kube-api-access-p6vl4\") pod \"alertmanager-main-0\" (UID: \"47233862-95a6-47dd-b064-4385f0aca30b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:27.059035 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:27.059005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 02 09:35:27.190543 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:27.190511 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 02 09:35:27.192458 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:35:27.192434 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47233862_95a6_47dd_b064_4385f0aca30b.slice/crio-ceeffb45b0b8108c628878a1762014e5010d643153939c7308e2b7fbc90cce65 WatchSource:0}: Error finding container ceeffb45b0b8108c628878a1762014e5010d643153939c7308e2b7fbc90cce65: Status 404 returned error can't find the container with id ceeffb45b0b8108c628878a1762014e5010d643153939c7308e2b7fbc90cce65 Aug 02 09:35:27.700407 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:27.700368 2564 generic.go:356] "Generic (PLEG): container finished" podID="47233862-95a6-47dd-b064-4385f0aca30b" containerID="3267a75e263f7a730198dd9a43d17d929a7dfa1d85781158270f89eeb59145e8" exitCode=0 Aug 02 09:35:27.700790 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:27.700423 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47233862-95a6-47dd-b064-4385f0aca30b","Type":"ContainerDied","Data":"3267a75e263f7a730198dd9a43d17d929a7dfa1d85781158270f89eeb59145e8"} Aug 02 09:35:27.700790 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:27.700541 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47233862-95a6-47dd-b064-4385f0aca30b","Type":"ContainerStarted","Data":"ceeffb45b0b8108c628878a1762014e5010d643153939c7308e2b7fbc90cce65"} Aug 02 09:35:28.471293 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.471243 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-64fd7c86d-cgsd6"] Aug 02 09:35:28.474284 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.474262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.477546 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.477443 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Aug 02 09:35:28.477739 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.477720 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Aug 02 09:35:28.477739 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.477730 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Aug 02 09:35:28.477873 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.477742 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Aug 02 09:35:28.477873 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.477749 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-k8cfh\"" Aug 02 09:35:28.477873 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.477720 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Aug 02 09:35:28.483912 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.483197 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Aug 02 09:35:28.487541 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.487519 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64fd7c86d-cgsd6"] Aug 02 09:35:28.588626 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.588592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-federate-client-tls\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.588790 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.588685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-metrics-client-ca\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.588790 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.588712 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-telemeter-client-tls\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.588918 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.588894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njpcf\" (UniqueName: \"kubernetes.io/projected/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-kube-api-access-njpcf\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.588987 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.588967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.589071 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.589055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-serving-certs-ca-bundle\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.589150 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.589133 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-secret-telemeter-client\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.589211 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.589190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.689958 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.689914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-serving-certs-ca-bundle\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.690140 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.689965 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-secret-telemeter-client\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.690140 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.689998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.690140 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.690029 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-federate-client-tls\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.690140 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.690052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-metrics-client-ca\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.690140 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.690075 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-telemeter-client-tls\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.690390 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.690165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njpcf\" (UniqueName: \"kubernetes.io/projected/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-kube-api-access-njpcf\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.690390 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.690224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.691600 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.691573 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.691777 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.691663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-metrics-client-ca\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.691888 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.691759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-serving-certs-ca-bundle\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.693430 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.693398 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-secret-telemeter-client\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.693655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.693628 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-telemeter-client-tls\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.693743 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.693717 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.693797 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.693787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-federate-client-tls\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.697964 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.697941 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njpcf\" (UniqueName: \"kubernetes.io/projected/cfd25b47-3ce0-48ea-a491-fd08ebb6a523-kube-api-access-njpcf\") pod \"telemeter-client-64fd7c86d-cgsd6\" (UID: \"cfd25b47-3ce0-48ea-a491-fd08ebb6a523\") " pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.708244 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.708215 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47233862-95a6-47dd-b064-4385f0aca30b","Type":"ContainerStarted","Data":"d341f861a582ba76356cb0eb2fb0ce17355cdbdce17d37855d44f4e8aae4fe1d"} Aug 02 09:35:28.708595 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.708288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47233862-95a6-47dd-b064-4385f0aca30b","Type":"ContainerStarted","Data":"1aeaae6efca8e70525cc47f53ff0dbfed28f7a1908d15580f032725d19be691d"} Aug 02 09:35:28.708595 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.708320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47233862-95a6-47dd-b064-4385f0aca30b","Type":"ContainerStarted","Data":"d418e5e2ed5088c3caf74c11508d115d307372b6bbd6faf2227898e45c4925aa"} Aug 02 09:35:28.708595 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.708362 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47233862-95a6-47dd-b064-4385f0aca30b","Type":"ContainerStarted","Data":"4c2becd607ebd639cee0aff18f572b4d629c83ba6d4b65db046d9c080c68d97d"} Aug 02 09:35:28.708595 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.708392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47233862-95a6-47dd-b064-4385f0aca30b","Type":"ContainerStarted","Data":"c442ec7f34319d60881e5d877aef468cd98aa361898f80c3643dfa2624276197"} Aug 02 09:35:28.708595 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.708431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"47233862-95a6-47dd-b064-4385f0aca30b","Type":"ContainerStarted","Data":"9fd28112aa47db935790946ba632f1e9d4ca1016794cc6de04e85be3c13439c6"} Aug 02 09:35:28.733529 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.733443 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.718451273 podStartE2EDuration="2.73342559s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.974317ms" isStatefulPod=false podCreationTimestamp="2026-08-02 09:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:35:28.7311361 +0000 UTC m=+150.172026558" watchObservedRunningTime="2026-08-02 09:35:28.73342559 +0000 UTC m=+150.174316046" Aug 02 09:35:28.764402 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.764359 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 02 09:35:28.764790 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.764751 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="prometheus" containerID="cri-o://1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005" gracePeriod=600 Aug 02 09:35:28.765142 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.765113 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy-thanos" containerID="cri-o://769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e" gracePeriod=600 Aug 02 09:35:28.765242 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.765224 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy" containerID="cri-o://361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7" gracePeriod=600 Aug 02 09:35:28.765320 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.765304 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy-web" containerID="cri-o://003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257" gracePeriod=600 Aug 02 09:35:28.765397 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.765381 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="thanos-sidecar" containerID="cri-o://3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe" gracePeriod=600 Aug 02 09:35:28.765491 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.765456 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="config-reloader" containerID="cri-o://978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c" gracePeriod=600 Aug 02 09:35:28.787436 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.787223 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" Aug 02 09:35:28.963616 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:28.963582 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64fd7c86d-cgsd6"] Aug 02 09:35:28.967757 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:35:28.967730 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd25b47_3ce0_48ea_a491_fd08ebb6a523.slice/crio-ecaeb722d70ec19ad256adec452327f65307c469feb6de65fca331bb741f40a2 WatchSource:0}: Error finding container ecaeb722d70ec19ad256adec452327f65307c469feb6de65fca331bb741f40a2: Status 404 returned error can't find the container with id ecaeb722d70ec19ad256adec452327f65307c469feb6de65fca331bb741f40a2 Aug 02 09:35:29.035060 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.035028 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.096891 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.096859 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-metrics-client-ca\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.096891 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.096895 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-thanos-prometheus-http-client-file\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097119 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.096920 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4grrc\" (UniqueName: \"kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-kube-api-access-4grrc\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097119 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.096946 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-0\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097119 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.096996 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097119 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097051 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-trusted-ca-bundle\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097119 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097085 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-grpc-tls\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097119 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097107 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-2\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097374 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097145 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097374 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097180 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-metrics-client-certs\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097374 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097206 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-kubelet-serving-ca-bundle\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097374 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097229 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-config\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097374 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097260 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-config-out\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097374 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097287 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-web-config\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097374 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097319 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-1\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097374 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097340 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-tls-assets\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097374 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097370 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-serving-certs-ca-bundle\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097830 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097408 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-tls\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097830 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097431 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-kube-rbac-proxy\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097830 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097476 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-db\") pod \"f036671c-4718-4c1c-8384-7c350be8a8fe\" (UID: \"f036671c-4718-4c1c-8384-7c350be8a8fe\") " Aug 02 09:35:29.097830 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097512 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 09:35:29.097830 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097774 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.098066 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.097933 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 09:35:29.098177 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.098068 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 09:35:29.098897 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.098866 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 09:35:29.099022 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.099001 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:35:29.099425 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.099402 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 09:35:29.099969 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.099942 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:29.100603 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.100303 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:29.100959 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.100824 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-kube-api-access-4grrc" (OuterVolumeSpecName: "kube-api-access-4grrc") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "kube-api-access-4grrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 09:35:29.101628 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.101358 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 09:35:29.102277 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.101829 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 09:35:29.102277 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.102184 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 09:35:29.102277 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.102250 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:29.102623 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.102588 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-config-out" (OuterVolumeSpecName: "config-out") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:35:29.102830 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.102784 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:29.102976 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.102945 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:29.103136 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.103104 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:29.103599 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.103578 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:29.103740 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.103717 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-config" (OuterVolumeSpecName: "config") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:29.115737 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.115709 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-web-config" (OuterVolumeSpecName: "web-config") pod "f036671c-4718-4c1c-8384-7c350be8a8fe" (UID: "f036671c-4718-4c1c-8384-7c350be8a8fe"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:35:29.198371 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198340 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-db\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198371 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198367 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-metrics-client-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198371 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198378 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198387 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4grrc\" (UniqueName: \"kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-kube-api-access-4grrc\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198395 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198404 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198413 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198421 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-grpc-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198430 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198438 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-metrics-client-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198447 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198455 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198483 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f036671c-4718-4c1c-8384-7c350be8a8fe-config-out\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198493 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-web-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198501 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198508 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f036671c-4718-4c1c-8384-7c350be8a8fe-tls-assets\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198516 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f036671c-4718-4c1c-8384-7c350be8a8fe-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198523 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.198605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.198531 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f036671c-4718-4c1c-8384-7c350be8a8fe-secret-kube-rbac-proxy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:35:29.713758 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.713710 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" event={"ID":"cfd25b47-3ce0-48ea-a491-fd08ebb6a523","Type":"ContainerStarted","Data":"ecaeb722d70ec19ad256adec452327f65307c469feb6de65fca331bb741f40a2"} Aug 02 09:35:29.717349 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717325 2564 generic.go:356] "Generic (PLEG): container finished" podID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerID="769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e" exitCode=0 Aug 02 09:35:29.717349 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717348 2564 generic.go:356] "Generic (PLEG): container finished" podID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerID="361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7" exitCode=0 Aug 02 09:35:29.717349 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717355 2564 generic.go:356] "Generic (PLEG): container finished" podID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerID="003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257" exitCode=0 Aug 02 09:35:29.717627 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717360 2564 generic.go:356] "Generic (PLEG): container finished" podID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerID="3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe" exitCode=0 Aug 02 09:35:29.717627 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717368 2564 generic.go:356] "Generic (PLEG): container finished" podID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerID="978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c" exitCode=0 Aug 02 09:35:29.717627 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717376 2564 generic.go:356] "Generic (PLEG): container finished" podID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerID="1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005" exitCode=0 Aug 02 09:35:29.717627 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717490 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.717627 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerDied","Data":"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e"} Aug 02 09:35:29.717627 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerDied","Data":"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7"} Aug 02 09:35:29.717627 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717605 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerDied","Data":"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257"} Aug 02 09:35:29.718035 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717638 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerDied","Data":"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe"} Aug 02 09:35:29.718035 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717657 2564 scope.go:117] "RemoveContainer" containerID="769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e" Aug 02 09:35:29.718035 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerDied","Data":"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c"} Aug 02 09:35:29.718035 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerDied","Data":"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005"} Aug 02 09:35:29.718035 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.717761 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f036671c-4718-4c1c-8384-7c350be8a8fe","Type":"ContainerDied","Data":"773a27f19e30ce091351310f57e1fb37cd2df9d3672e64d7e41aeb13c4b6269d"} Aug 02 09:35:29.728192 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.728174 2564 scope.go:117] "RemoveContainer" containerID="361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7" Aug 02 09:35:29.736449 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.736428 2564 scope.go:117] "RemoveContainer" containerID="003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257" Aug 02 09:35:29.742311 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.742291 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 02 09:35:29.746662 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.746639 2564 scope.go:117] "RemoveContainer" containerID="3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe" Aug 02 09:35:29.747748 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.747728 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 02 09:35:29.756072 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.756052 2564 scope.go:117] "RemoveContainer" containerID="978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c" Aug 02 09:35:29.763834 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.763818 2564 scope.go:117] "RemoveContainer" containerID="1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005" Aug 02 09:35:29.771540 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771517 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 02 09:35:29.771888 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771874 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy-thanos" Aug 02 09:35:29.771950 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771890 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy-thanos" Aug 02 09:35:29.771950 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771910 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="config-reloader" Aug 02 09:35:29.771950 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771919 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="config-reloader" Aug 02 09:35:29.771950 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771932 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="thanos-sidecar" Aug 02 09:35:29.771950 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771941 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="thanos-sidecar" Aug 02 09:35:29.772114 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771954 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="prometheus" Aug 02 09:35:29.772114 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771963 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="prometheus" Aug 02 09:35:29.772114 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771972 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy-web" Aug 02 09:35:29.772114 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771980 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy-web" Aug 02 09:35:29.772114 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771990 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy" Aug 02 09:35:29.772114 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.771998 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy" Aug 02 09:35:29.772114 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.772013 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="init-config-reloader" Aug 02 09:35:29.772114 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.772022 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="init-config-reloader" Aug 02 09:35:29.772114 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.772108 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="prometheus" Aug 02 09:35:29.772544 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.772121 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="config-reloader" Aug 02 09:35:29.772544 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.772131 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="thanos-sidecar" Aug 02 09:35:29.772544 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.772140 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy-thanos" Aug 02 09:35:29.772544 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.772150 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy-web" Aug 02 09:35:29.772544 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.772158 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" containerName="kube-rbac-proxy" Aug 02 09:35:29.773038 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.772760 2564 scope.go:117] "RemoveContainer" containerID="a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4" Aug 02 09:35:29.775928 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.775908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.778640 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.778622 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 02 09:35:29.778819 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.778637 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lqcrs\"" Aug 02 09:35:29.778819 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.778647 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 02 09:35:29.779146 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.779070 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 02 09:35:29.779146 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.779098 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 02 09:35:29.779435 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.779418 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 02 09:35:29.779649 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.779631 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 02 09:35:29.779721 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.779600 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 02 09:35:29.779721 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.779674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a9uus8plrndk3\"" Aug 02 09:35:29.780101 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.780082 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 02 09:35:29.780205 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.780188 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 02 09:35:29.780328 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.780312 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 02 09:35:29.780391 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.780354 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 02 09:35:29.780809 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.780792 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 02 09:35:29.780993 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.780977 2564 scope.go:117] "RemoveContainer" containerID="769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e" Aug 02 09:35:29.781308 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:35:29.781279 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": container with ID starting with 769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e not found: ID does not exist" containerID="769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e" Aug 02 09:35:29.781394 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.781317 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e"} err="failed to get container status \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": rpc error: code = NotFound desc = could not find container \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": container with ID starting with 769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e not found: ID does not exist" Aug 02 09:35:29.781394 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.781358 2564 scope.go:117] "RemoveContainer" containerID="361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7" Aug 02 09:35:29.781666 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:35:29.781644 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": container with ID starting with 361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7 not found: ID does not exist" containerID="361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7" Aug 02 09:35:29.781728 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.781675 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7"} err="failed to get container status \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": rpc error: code = NotFound desc = could not find container \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": container with ID starting with 361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7 not found: ID does not exist" Aug 02 09:35:29.781728 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.781697 2564 scope.go:117] "RemoveContainer" containerID="003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257" Aug 02 09:35:29.781971 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:35:29.781950 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": container with ID starting with 003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257 not found: ID does not exist" containerID="003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257" Aug 02 09:35:29.782012 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.781981 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257"} err="failed to get container status \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": rpc error: code = NotFound desc = could not find container \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": container with ID starting with 003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257 not found: ID does not exist" Aug 02 09:35:29.782012 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.782004 2564 scope.go:117] "RemoveContainer" containerID="3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe" Aug 02 09:35:29.782254 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:35:29.782234 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": container with ID starting with 3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe not found: ID does not exist" containerID="3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe" Aug 02 09:35:29.782301 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.782262 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe"} err="failed to get container status \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": rpc error: code = NotFound desc = could not find container \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": container with ID starting with 3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe not found: ID does not exist" Aug 02 09:35:29.782301 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.782282 2564 scope.go:117] "RemoveContainer" containerID="978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c" Aug 02 09:35:29.782556 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:35:29.782533 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": container with ID starting with 978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c not found: ID does not exist" containerID="978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c" Aug 02 09:35:29.782640 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.782563 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c"} err="failed to get container status \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": rpc error: code = NotFound desc = could not find container \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": container with ID starting with 978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c not found: ID does not exist" Aug 02 09:35:29.782640 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.782591 2564 scope.go:117] "RemoveContainer" containerID="1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005" Aug 02 09:35:29.782833 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:35:29.782813 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": container with ID starting with 1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005 not found: ID does not exist" containerID="1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005" Aug 02 09:35:29.782901 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.782834 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005"} err="failed to get container status \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": rpc error: code = NotFound desc = could not find container \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": container with ID starting with 1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005 not found: ID does not exist" Aug 02 09:35:29.782901 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.782852 2564 scope.go:117] "RemoveContainer" containerID="a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4" Aug 02 09:35:29.782901 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.782862 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 02 09:35:29.783183 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:35:29.783166 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": container with ID starting with a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4 not found: ID does not exist" containerID="a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4" Aug 02 09:35:29.783229 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.783186 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4"} err="failed to get container status \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": rpc error: code = NotFound desc = could not find container \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": container with ID starting with a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4 not found: ID does not exist" Aug 02 09:35:29.783229 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.783202 2564 scope.go:117] "RemoveContainer" containerID="769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e" Aug 02 09:35:29.783420 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.783403 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e"} err="failed to get container status \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": rpc error: code = NotFound desc = could not find container \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": container with ID starting with 769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e not found: ID does not exist" Aug 02 09:35:29.783492 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.783420 2564 scope.go:117] "RemoveContainer" containerID="361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7" Aug 02 09:35:29.783740 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.783704 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7"} err="failed to get container status \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": rpc error: code = NotFound desc = could not find container \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": container with ID starting with 361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7 not found: ID does not exist" Aug 02 09:35:29.783740 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.783729 2564 scope.go:117] "RemoveContainer" containerID="003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257" Aug 02 09:35:29.784003 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.783981 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257"} err="failed to get container status \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": rpc error: code = NotFound desc = could not find container \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": container with ID starting with 003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257 not found: ID does not exist" Aug 02 09:35:29.784084 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.784003 2564 scope.go:117] "RemoveContainer" containerID="3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe" Aug 02 09:35:29.784231 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.784207 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe"} err="failed to get container status \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": rpc error: code = NotFound desc = could not find container \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": container with ID starting with 3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe not found: ID does not exist" Aug 02 09:35:29.784277 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.784232 2564 scope.go:117] "RemoveContainer" containerID="978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c" Aug 02 09:35:29.784489 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.784444 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c"} err="failed to get container status \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": rpc error: code = NotFound desc = could not find container \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": container with ID starting with 978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c not found: ID does not exist" Aug 02 09:35:29.784576 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.784488 2564 scope.go:117] "RemoveContainer" containerID="1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005" Aug 02 09:35:29.784576 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.784531 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 02 09:35:29.784717 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.784691 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005"} err="failed to get container status \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": rpc error: code = NotFound desc = could not find container \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": container with ID starting with 1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005 not found: ID does not exist" Aug 02 09:35:29.784776 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.784719 2564 scope.go:117] "RemoveContainer" containerID="a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4" Aug 02 09:35:29.784942 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.784918 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4"} err="failed to get container status \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": rpc error: code = NotFound desc = could not find container \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": container with ID starting with a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4 not found: ID does not exist" Aug 02 09:35:29.785005 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.784945 2564 scope.go:117] "RemoveContainer" containerID="769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e" Aug 02 09:35:29.785307 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.785222 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e"} err="failed to get container status \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": rpc error: code = NotFound desc = could not find container \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": container with ID starting with 769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e not found: ID does not exist" Aug 02 09:35:29.785307 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.785247 2564 scope.go:117] "RemoveContainer" containerID="361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7" Aug 02 09:35:29.785550 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.785530 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7"} err="failed to get container status \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": rpc error: code = NotFound desc = could not find container \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": container with ID starting with 361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7 not found: ID does not exist" Aug 02 09:35:29.785550 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.785549 2564 scope.go:117] "RemoveContainer" containerID="003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257" Aug 02 09:35:29.785787 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.785766 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257"} err="failed to get container status \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": rpc error: code = NotFound desc = could not find container \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": container with ID starting with 003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257 not found: ID does not exist" Aug 02 09:35:29.785787 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.785786 2564 scope.go:117] "RemoveContainer" containerID="3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe" Aug 02 09:35:29.786039 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.786018 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe"} err="failed to get container status \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": rpc error: code = NotFound desc = could not find container \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": container with ID starting with 3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe not found: ID does not exist" Aug 02 09:35:29.786096 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.786042 2564 scope.go:117] "RemoveContainer" containerID="978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c" Aug 02 09:35:29.786245 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.786224 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c"} err="failed to get container status \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": rpc error: code = NotFound desc = could not find container \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": container with ID starting with 978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c not found: ID does not exist" Aug 02 09:35:29.786245 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.786243 2564 scope.go:117] "RemoveContainer" containerID="1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005" Aug 02 09:35:29.786430 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.786411 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005"} err="failed to get container status \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": rpc error: code = NotFound desc = could not find container \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": container with ID starting with 1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005 not found: ID does not exist" Aug 02 09:35:29.786430 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.786429 2564 scope.go:117] "RemoveContainer" containerID="a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4" Aug 02 09:35:29.786643 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.786625 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4"} err="failed to get container status \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": rpc error: code = NotFound desc = could not find container \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": container with ID starting with a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4 not found: ID does not exist" Aug 02 09:35:29.786643 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.786641 2564 scope.go:117] "RemoveContainer" containerID="769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e" Aug 02 09:35:29.786823 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.786796 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e"} err="failed to get container status \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": rpc error: code = NotFound desc = could not find container \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": container with ID starting with 769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e not found: ID does not exist" Aug 02 09:35:29.786823 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.786821 2564 scope.go:117] "RemoveContainer" containerID="361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7" Aug 02 09:35:29.787009 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.786993 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7"} err="failed to get container status \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": rpc error: code = NotFound desc = could not find container \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": container with ID starting with 361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7 not found: ID does not exist" Aug 02 09:35:29.787009 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.787008 2564 scope.go:117] "RemoveContainer" containerID="003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257" Aug 02 09:35:29.787218 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.787198 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257"} err="failed to get container status \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": rpc error: code = NotFound desc = could not find container \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": container with ID starting with 003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257 not found: ID does not exist" Aug 02 09:35:29.787218 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.787218 2564 scope.go:117] "RemoveContainer" containerID="3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe" Aug 02 09:35:29.787416 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.787396 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe"} err="failed to get container status \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": rpc error: code = NotFound desc = could not find container \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": container with ID starting with 3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe not found: ID does not exist" Aug 02 09:35:29.787486 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.787419 2564 scope.go:117] "RemoveContainer" containerID="978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c" Aug 02 09:35:29.787674 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.787655 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c"} err="failed to get container status \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": rpc error: code = NotFound desc = could not find container \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": container with ID starting with 978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c not found: ID does not exist" Aug 02 09:35:29.787742 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.787675 2564 scope.go:117] "RemoveContainer" containerID="1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005" Aug 02 09:35:29.787918 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.787901 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005"} err="failed to get container status \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": rpc error: code = NotFound desc = could not find container \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": container with ID starting with 1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005 not found: ID does not exist" Aug 02 09:35:29.787960 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.787920 2564 scope.go:117] "RemoveContainer" containerID="a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4" Aug 02 09:35:29.788136 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.788119 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4"} err="failed to get container status \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": rpc error: code = NotFound desc = could not find container \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": container with ID starting with a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4 not found: ID does not exist" Aug 02 09:35:29.788187 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.788136 2564 scope.go:117] "RemoveContainer" containerID="769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e" Aug 02 09:35:29.788373 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.788356 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e"} err="failed to get container status \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": rpc error: code = NotFound desc = could not find container \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": container with ID starting with 769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e not found: ID does not exist" Aug 02 09:35:29.788373 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.788372 2564 scope.go:117] "RemoveContainer" containerID="361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7" Aug 02 09:35:29.788630 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.788611 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7"} err="failed to get container status \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": rpc error: code = NotFound desc = could not find container \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": container with ID starting with 361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7 not found: ID does not exist" Aug 02 09:35:29.788762 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.788632 2564 scope.go:117] "RemoveContainer" containerID="003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257" Aug 02 09:35:29.789216 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.789184 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257"} err="failed to get container status \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": rpc error: code = NotFound desc = could not find container \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": container with ID starting with 003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257 not found: ID does not exist" Aug 02 09:35:29.789216 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.789216 2564 scope.go:117] "RemoveContainer" containerID="3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe" Aug 02 09:35:29.789531 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.789511 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe"} err="failed to get container status \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": rpc error: code = NotFound desc = could not find container \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": container with ID starting with 3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe not found: ID does not exist" Aug 02 09:35:29.789597 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.789532 2564 scope.go:117] "RemoveContainer" containerID="978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c" Aug 02 09:35:29.789786 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.789767 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c"} err="failed to get container status \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": rpc error: code = NotFound desc = could not find container \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": container with ID starting with 978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c not found: ID does not exist" Aug 02 09:35:29.789917 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.789789 2564 scope.go:117] "RemoveContainer" containerID="1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005" Aug 02 09:35:29.790041 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.790017 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005"} err="failed to get container status \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": rpc error: code = NotFound desc = could not find container \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": container with ID starting with 1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005 not found: ID does not exist" Aug 02 09:35:29.790123 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.790043 2564 scope.go:117] "RemoveContainer" containerID="a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4" Aug 02 09:35:29.790298 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.790280 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4"} err="failed to get container status \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": rpc error: code = NotFound desc = could not find container \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": container with ID starting with a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4 not found: ID does not exist" Aug 02 09:35:29.790342 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.790300 2564 scope.go:117] "RemoveContainer" containerID="769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e" Aug 02 09:35:29.790532 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.790511 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e"} err="failed to get container status \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": rpc error: code = NotFound desc = could not find container \"769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e\": container with ID starting with 769eaf224a936ac5ec39cdfab89f0a5146ee6e36a06b817a7c1173b430003c1e not found: ID does not exist" Aug 02 09:35:29.790532 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.790531 2564 scope.go:117] "RemoveContainer" containerID="361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7" Aug 02 09:35:29.790791 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.790771 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7"} err="failed to get container status \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": rpc error: code = NotFound desc = could not find container \"361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7\": container with ID starting with 361e39ba36a545468a653aec3a8dfefa6de453057ea37e1065efaffdee4476e7 not found: ID does not exist" Aug 02 09:35:29.790852 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.790793 2564 scope.go:117] "RemoveContainer" containerID="003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257" Aug 02 09:35:29.791032 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.791010 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257"} err="failed to get container status \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": rpc error: code = NotFound desc = could not find container \"003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257\": container with ID starting with 003243edb94845715032860386cede732aa9d8a259ad437d4ed9b9c54e132257 not found: ID does not exist" Aug 02 09:35:29.791091 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.791033 2564 scope.go:117] "RemoveContainer" containerID="3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe" Aug 02 09:35:29.791239 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.791221 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 02 09:35:29.791279 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.791262 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe"} err="failed to get container status \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": rpc error: code = NotFound desc = could not find container \"3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe\": container with ID starting with 3c5a4f3ec1d146d8ed7667823232b3604e2e6857eed2e622761cdb89c2254dbe not found: ID does not exist" Aug 02 09:35:29.791324 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.791281 2564 scope.go:117] "RemoveContainer" containerID="978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c" Aug 02 09:35:29.791540 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.791523 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c"} err="failed to get container status \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": rpc error: code = NotFound desc = could not find container \"978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c\": container with ID starting with 978bb1e113b95febc48d6aa828b04bfa9bcd6ee1133ac05da206fbfc37b00e4c not found: ID does not exist" Aug 02 09:35:29.791578 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.791542 2564 scope.go:117] "RemoveContainer" containerID="1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005" Aug 02 09:35:29.791759 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.791742 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005"} err="failed to get container status \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": rpc error: code = NotFound desc = could not find container \"1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005\": container with ID starting with 1c3f54e1957329379ec6c59960b38378f28ce97c3ceb43e8c3c6569a93c72005 not found: ID does not exist" Aug 02 09:35:29.791803 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.791761 2564 scope.go:117] "RemoveContainer" containerID="a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4" Aug 02 09:35:29.791991 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.791976 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4"} err="failed to get container status \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": rpc error: code = NotFound desc = could not find container \"a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4\": container with ID starting with a01eafdd0568e64a9fbdd68fd4c37d60b9d9db4574066a753031bf825e5144d4 not found: ID does not exist" Aug 02 09:35:29.804561 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.804499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.804561 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.804536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.804695 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.804564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8ln\" (UniqueName: \"kubernetes.io/projected/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-kube-api-access-rz8ln\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.804695 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.804657 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.804795 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.804771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805248 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.804967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805248 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805248 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-config-out\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805248 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805248 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805248 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805663 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-web-config\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805663 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805663 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805663 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805380 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-config\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805663 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805663 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805663 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.805663 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805580 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.806122 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.805645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.906647 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.906602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-config-out\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.906647 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.906649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.906884 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.906775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.906884 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.906822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.906884 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.906849 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-web-config\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.906884 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.906874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907100 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.906902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907100 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.906929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-config\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907100 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.906968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907100 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.906997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907100 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907100 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907100 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907486 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907486 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907486 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8ln\" (UniqueName: \"kubernetes.io/projected/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-kube-api-access-rz8ln\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907486 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907486 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907486 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907486 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907805 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.907805 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.907657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.908840 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.908816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.909330 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.909255 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.909865 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.909840 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.909980 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.909895 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-config-out\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.911159 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.910702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.911159 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.910988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.911159 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.911004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-config\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.912749 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.912223 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.912749 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.912406 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.913372 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.913321 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-web-config\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.917212 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.915457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.919348 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.919307 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.919348 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.919325 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.921113 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.921068 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.921113 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.921089 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.921113 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.921107 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.921342 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.921130 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:29.924667 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:29.924644 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8ln\" (UniqueName: \"kubernetes.io/projected/13dc4b27-5c2e-4a0b-88bb-47dcc0746435-kube-api-access-rz8ln\") pod \"prometheus-k8s-0\" (UID: \"13dc4b27-5c2e-4a0b-88bb-47dcc0746435\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:30.088747 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:30.088665 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:35:30.237627 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:30.237599 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 02 09:35:30.240086 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:35:30.240056 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dc4b27_5c2e_4a0b_88bb_47dcc0746435.slice/crio-d9eebbe5a45b50c72dc4d5924f82b09acd19d81eee000b0fd0143930f8e8822c WatchSource:0}: Error finding container d9eebbe5a45b50c72dc4d5924f82b09acd19d81eee000b0fd0143930f8e8822c: Status 404 returned error can't find the container with id d9eebbe5a45b50c72dc4d5924f82b09acd19d81eee000b0fd0143930f8e8822c Aug 02 09:35:30.725329 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:30.725059 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13dc4b27-5c2e-4a0b-88bb-47dcc0746435","Type":"ContainerStarted","Data":"cab6d8300f070cf91010946a08ce72d9c3bad704b50aca27aad117f333708187"} Aug 02 09:35:30.725329 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:30.725133 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13dc4b27-5c2e-4a0b-88bb-47dcc0746435","Type":"ContainerStarted","Data":"d9eebbe5a45b50c72dc4d5924f82b09acd19d81eee000b0fd0143930f8e8822c"} Aug 02 09:35:30.978272 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:30.978217 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f036671c-4718-4c1c-8384-7c350be8a8fe" path="/var/lib/kubelet/pods/f036671c-4718-4c1c-8384-7c350be8a8fe/volumes" Aug 02 09:35:31.732000 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:31.731967 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" event={"ID":"cfd25b47-3ce0-48ea-a491-fd08ebb6a523","Type":"ContainerStarted","Data":"4989b7ada0be34361c8b803a85eb95027e1e0b126f840ed9c9fa9dbc97d9e13a"} Aug 02 09:35:31.732418 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:31.732034 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" event={"ID":"cfd25b47-3ce0-48ea-a491-fd08ebb6a523","Type":"ContainerStarted","Data":"254c014e948b9d5a94cdd4525fc71c7354a2df6112a133999c40fc0b19295467"} Aug 02 09:35:31.732418 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:31.732067 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" event={"ID":"cfd25b47-3ce0-48ea-a491-fd08ebb6a523","Type":"ContainerStarted","Data":"b3fc44085459f8b17ee7c8c09e1effe65291715cdc77a8dc3043b45382eea759"} Aug 02 09:35:31.733432 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:31.733403 2564 generic.go:356] "Generic (PLEG): container finished" podID="13dc4b27-5c2e-4a0b-88bb-47dcc0746435" containerID="cab6d8300f070cf91010946a08ce72d9c3bad704b50aca27aad117f333708187" exitCode=0 Aug 02 09:35:31.733557 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:31.733491 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13dc4b27-5c2e-4a0b-88bb-47dcc0746435","Type":"ContainerDied","Data":"cab6d8300f070cf91010946a08ce72d9c3bad704b50aca27aad117f333708187"} Aug 02 09:35:31.754932 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:31.754891 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-64fd7c86d-cgsd6" podStartSLOduration=1.9167702599999998 podStartE2EDuration="3.754876904s" totalImagesPullingTime="1.838106644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:35:31.753575252 +0000 UTC m=+153.194465708" watchObservedRunningTime="2026-08-02 09:35:31.754876904 +0000 UTC m=+153.195767372" Aug 02 09:35:32.742079 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:32.741974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13dc4b27-5c2e-4a0b-88bb-47dcc0746435","Type":"ContainerStarted","Data":"6f45ad4c9d43e435e1cadd59ad1d125f213840339236d63568f446826a7c8c33"} Aug 02 09:35:32.742531 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:32.742100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13dc4b27-5c2e-4a0b-88bb-47dcc0746435","Type":"ContainerStarted","Data":"e78d260885ae7e7c28255c3ef87471037dcf240a5d7b055eda3a2dbe92aa2003"} Aug 02 09:35:32.742531 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:32.742145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13dc4b27-5c2e-4a0b-88bb-47dcc0746435","Type":"ContainerStarted","Data":"be2cfadb7ba3a22bb8e54c7ef819f43724a9dd374a679d51786edc146e44b139"} Aug 02 09:35:32.742531 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:32.742182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13dc4b27-5c2e-4a0b-88bb-47dcc0746435","Type":"ContainerStarted","Data":"d744bd7ab20fc507870a64b1a940a69fe72f5b7f876b5f4d948236bb97f3226e"} Aug 02 09:35:32.742531 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:32.742222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13dc4b27-5c2e-4a0b-88bb-47dcc0746435","Type":"ContainerStarted","Data":"8ac2dc23ac5605cb737cb4a8fd52cfe516b16d5ce3b3e002dfcb2b5e801b8707"} Aug 02 09:35:32.742531 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:32.742258 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"13dc4b27-5c2e-4a0b-88bb-47dcc0746435","Type":"ContainerStarted","Data":"da11ca438cb4041f54f5ea740aa8ed4eaec95b377f7585772fb3dcfed37db8d1"} Aug 02 09:35:32.767292 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:32.767240 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.741328793 podStartE2EDuration="3.767218316s" totalImagesPullingTime="0s" totalInitContainerRuntime="25.889523ms" isStatefulPod=false podCreationTimestamp="2026-08-02 09:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:35:32.7653089 +0000 UTC m=+154.206199397" watchObservedRunningTime="2026-08-02 09:35:32.767218316 +0000 UTC m=+154.208108773" Aug 02 09:35:35.089405 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:35:35.089371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:36:30.089295 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:36:30.089259 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:36:30.104929 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:36:30.104907 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:36:30.981653 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:36:30.981628 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 02 09:37:58.899087 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:37:58.899061 2564 kubelet.go:1648] "Image garbage collection succeeded" Aug 02 09:38:15.707932 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.707898 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2"] Aug 02 09:38:15.711254 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.711234 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2" Aug 02 09:38:15.713611 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.713584 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 02 09:38:15.713743 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.713650 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 02 09:38:15.714339 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.714324 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-44sj7\"" Aug 02 09:38:15.718558 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.718380 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2"] Aug 02 09:38:15.847926 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.847893 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9579\" (UniqueName: \"kubernetes.io/projected/ede0eb4f-0b37-46ad-be73-1bb9a8374afc-kube-api-access-v9579\") pod \"cert-manager-cainjector-559cdd8c87-kj9c2\" (UID: \"ede0eb4f-0b37-46ad-be73-1bb9a8374afc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2" Aug 02 09:38:15.848100 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.847938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ede0eb4f-0b37-46ad-be73-1bb9a8374afc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kj9c2\" (UID: \"ede0eb4f-0b37-46ad-be73-1bb9a8374afc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2" Aug 02 09:38:15.948929 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.948898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ede0eb4f-0b37-46ad-be73-1bb9a8374afc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kj9c2\" (UID: \"ede0eb4f-0b37-46ad-be73-1bb9a8374afc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2" Aug 02 09:38:15.949054 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.948992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9579\" (UniqueName: \"kubernetes.io/projected/ede0eb4f-0b37-46ad-be73-1bb9a8374afc-kube-api-access-v9579\") pod \"cert-manager-cainjector-559cdd8c87-kj9c2\" (UID: \"ede0eb4f-0b37-46ad-be73-1bb9a8374afc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2" Aug 02 09:38:15.957165 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.957140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ede0eb4f-0b37-46ad-be73-1bb9a8374afc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kj9c2\" (UID: \"ede0eb4f-0b37-46ad-be73-1bb9a8374afc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2" Aug 02 09:38:15.957340 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:15.957319 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9579\" (UniqueName: \"kubernetes.io/projected/ede0eb4f-0b37-46ad-be73-1bb9a8374afc-kube-api-access-v9579\") pod \"cert-manager-cainjector-559cdd8c87-kj9c2\" (UID: \"ede0eb4f-0b37-46ad-be73-1bb9a8374afc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2" Aug 02 09:38:16.032717 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:16.032636 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2" Aug 02 09:38:16.161976 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:16.161944 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2"] Aug 02 09:38:16.165768 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:38:16.165735 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podede0eb4f_0b37_46ad_be73_1bb9a8374afc.slice/crio-0cb42c63b5972d0dc619318a83a22eb6d5829032fb363b59086c857aa9528da0 WatchSource:0}: Error finding container 0cb42c63b5972d0dc619318a83a22eb6d5829032fb363b59086c857aa9528da0: Status 404 returned error can't find the container with id 0cb42c63b5972d0dc619318a83a22eb6d5829032fb363b59086c857aa9528da0 Aug 02 09:38:16.168106 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:16.168089 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 02 09:38:16.373695 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:16.373624 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2" event={"ID":"ede0eb4f-0b37-46ad-be73-1bb9a8374afc","Type":"ContainerStarted","Data":"0cb42c63b5972d0dc619318a83a22eb6d5829032fb363b59086c857aa9528da0"} Aug 02 09:38:19.391692 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:19.391652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2" event={"ID":"ede0eb4f-0b37-46ad-be73-1bb9a8374afc","Type":"ContainerStarted","Data":"103c70b5124248499540439b2d6c9206efd64df16767fdd72ea63ed24ea6b1d0"} Aug 02 09:38:19.406373 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:19.406271 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kj9c2" podStartSLOduration=1.421143483 podStartE2EDuration="4.406254597s" totalImagesPullingTime="2.985111114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:38:19.404093403 +0000 UTC m=+320.844983859" watchObservedRunningTime="2026-08-02 09:38:19.406254597 +0000 UTC m=+320.847145052" Aug 02 09:38:53.860203 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.860120 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw"] Aug 02 09:38:53.864522 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.864499 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:53.866757 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.866733 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 02 09:38:53.867631 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.867616 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 02 09:38:53.867723 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.867702 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 02 09:38:53.867776 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.867702 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 02 09:38:53.867776 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.867738 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tsqrk\"" Aug 02 09:38:53.867949 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.867935 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 02 09:38:53.874518 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.874497 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw"] Aug 02 09:38:53.938155 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.938120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7a7ef85-18be-4bda-95c0-2541f54850db-metrics-cert\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:53.938155 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.938155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2cwt\" (UniqueName: \"kubernetes.io/projected/d7a7ef85-18be-4bda-95c0-2541f54850db-kube-api-access-p2cwt\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:53.938326 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.938237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7a7ef85-18be-4bda-95c0-2541f54850db-cert\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:53.938326 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:53.938261 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7a7ef85-18be-4bda-95c0-2541f54850db-manager-config\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:54.039125 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:54.039093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7a7ef85-18be-4bda-95c0-2541f54850db-cert\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:54.039255 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:54.039131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7a7ef85-18be-4bda-95c0-2541f54850db-manager-config\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:54.039255 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:54.039167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7a7ef85-18be-4bda-95c0-2541f54850db-metrics-cert\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:54.039255 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:54.039186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2cwt\" (UniqueName: \"kubernetes.io/projected/d7a7ef85-18be-4bda-95c0-2541f54850db-kube-api-access-p2cwt\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:54.039876 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:54.039858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7a7ef85-18be-4bda-95c0-2541f54850db-manager-config\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:54.041536 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:54.041514 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7a7ef85-18be-4bda-95c0-2541f54850db-cert\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:54.041642 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:54.041593 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7a7ef85-18be-4bda-95c0-2541f54850db-metrics-cert\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:54.047270 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:54.047247 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2cwt\" (UniqueName: \"kubernetes.io/projected/d7a7ef85-18be-4bda-95c0-2541f54850db-kube-api-access-p2cwt\") pod \"lws-controller-manager-df7456b48-j4kkw\" (UID: \"d7a7ef85-18be-4bda-95c0-2541f54850db\") " pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:54.174392 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:54.174325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:54.302376 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:54.302342 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw"] Aug 02 09:38:54.305745 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:38:54.305718 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a7ef85_18be_4bda_95c0_2541f54850db.slice/crio-3d2775f78296d85e33b26b9cd452793f4aff177ce7cb422cbfe0899211ce6a6a WatchSource:0}: Error finding container 3d2775f78296d85e33b26b9cd452793f4aff177ce7cb422cbfe0899211ce6a6a: Status 404 returned error can't find the container with id 3d2775f78296d85e33b26b9cd452793f4aff177ce7cb422cbfe0899211ce6a6a Aug 02 09:38:54.530330 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:54.530296 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" event={"ID":"d7a7ef85-18be-4bda-95c0-2541f54850db","Type":"ContainerStarted","Data":"3d2775f78296d85e33b26b9cd452793f4aff177ce7cb422cbfe0899211ce6a6a"} Aug 02 09:38:56.285730 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.285700 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp"] Aug 02 09:38:56.287968 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.287946 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.290394 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.290343 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 02 09:38:56.290394 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.290373 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 02 09:38:56.290601 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.290407 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m9s7z\"" Aug 02 09:38:56.290601 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.290425 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 02 09:38:56.290714 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.290697 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 02 09:38:56.305960 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.305940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp"] Aug 02 09:38:56.360719 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.360685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a144164b-d9cd-42d5-9c29-329bae2aa0ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.360883 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.360753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmdh\" (UniqueName: \"kubernetes.io/projected/a144164b-d9cd-42d5-9c29-329bae2aa0ef-kube-api-access-vcmdh\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.360883 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.360796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a144164b-d9cd-42d5-9c29-329bae2aa0ef-tmp\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.360997 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.360885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a144164b-d9cd-42d5-9c29-329bae2aa0ef-manifests\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.360997 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.360922 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a144164b-d9cd-42d5-9c29-329bae2aa0ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.462185 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.462150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a144164b-d9cd-42d5-9c29-329bae2aa0ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.462352 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.462203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmdh\" (UniqueName: \"kubernetes.io/projected/a144164b-d9cd-42d5-9c29-329bae2aa0ef-kube-api-access-vcmdh\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.462352 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.462237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a144164b-d9cd-42d5-9c29-329bae2aa0ef-tmp\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.462352 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.462287 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a144164b-d9cd-42d5-9c29-329bae2aa0ef-manifests\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.462352 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.462312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a144164b-d9cd-42d5-9c29-329bae2aa0ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.462830 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.462783 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a144164b-d9cd-42d5-9c29-329bae2aa0ef-manifests\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.465024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.464982 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a144164b-d9cd-42d5-9c29-329bae2aa0ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.465024 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.465012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a144164b-d9cd-42d5-9c29-329bae2aa0ef-tmp\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.465227 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.465197 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a144164b-d9cd-42d5-9c29-329bae2aa0ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.476678 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.476651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmdh\" (UniqueName: \"kubernetes.io/projected/a144164b-d9cd-42d5-9c29-329bae2aa0ef-kube-api-access-vcmdh\") pod \"opendatahub-operator-controller-manager-5694fb54d4-trvzp\" (UID: \"a144164b-d9cd-42d5-9c29-329bae2aa0ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.600754 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.600685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:38:56.740982 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:56.740946 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp"] Aug 02 09:38:56.742194 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:38:56.742168 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda144164b_d9cd_42d5_9c29_329bae2aa0ef.slice/crio-a7caabde83b0f20238e97db4042fa9fe246b2eddbbf0ed4f3ee4ccb88a01f655 WatchSource:0}: Error finding container a7caabde83b0f20238e97db4042fa9fe246b2eddbbf0ed4f3ee4ccb88a01f655: Status 404 returned error can't find the container with id a7caabde83b0f20238e97db4042fa9fe246b2eddbbf0ed4f3ee4ccb88a01f655 Aug 02 09:38:57.546868 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:57.546829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" event={"ID":"d7a7ef85-18be-4bda-95c0-2541f54850db","Type":"ContainerStarted","Data":"17403cc8e04521ee5c4e7b8cf8f4979ec5809da07bd031450467c9e12128765a"} Aug 02 09:38:57.547329 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:57.546895 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:38:57.554744 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:57.549696 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" event={"ID":"a144164b-d9cd-42d5-9c29-329bae2aa0ef","Type":"ContainerStarted","Data":"a7caabde83b0f20238e97db4042fa9fe246b2eddbbf0ed4f3ee4ccb88a01f655"} Aug 02 09:38:57.563920 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:38:57.563863 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" podStartSLOduration=2.216414845 podStartE2EDuration="4.563842783s" totalImagesPullingTime="2.347427938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:38:57.562630709 +0000 UTC m=+359.003521166" watchObservedRunningTime="2026-08-02 09:38:57.563842783 +0000 UTC m=+359.004733243" Aug 02 09:39:00.566205 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:00.566171 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" event={"ID":"a144164b-d9cd-42d5-9c29-329bae2aa0ef","Type":"ContainerStarted","Data":"d107d772ffd85c8d05f8f8b041476f406059e35063940d2a57d9a468c94c77fb"} Aug 02 09:39:01.572159 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:01.572121 2564 generic.go:356] "Generic (PLEG): container finished" podID="a144164b-d9cd-42d5-9c29-329bae2aa0ef" containerID="d107d772ffd85c8d05f8f8b041476f406059e35063940d2a57d9a468c94c77fb" exitCode=0 Aug 02 09:39:01.572598 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:01.572211 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" event={"ID":"a144164b-d9cd-42d5-9c29-329bae2aa0ef","Type":"ContainerDied","Data":"d107d772ffd85c8d05f8f8b041476f406059e35063940d2a57d9a468c94c77fb"} Aug 02 09:39:02.578266 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:02.578229 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" event={"ID":"a144164b-d9cd-42d5-9c29-329bae2aa0ef","Type":"ContainerStarted","Data":"269a14164f4754a7633a03e1b269d4cb4a54e7e535be7e0bc10ca5a4e802f3c0"} Aug 02 09:39:02.578634 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:02.578377 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:39:02.599813 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:02.599754 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" podStartSLOduration=2.31247145 podStartE2EDuration="6.599736473s" totalImagesPullingTime="3.295754873s" totalInitContainerRuntime="991.51015ms" isStatefulPod=false podCreationTimestamp="2026-08-02 09:38:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:39:02.598912662 +0000 UTC m=+364.039803120" watchObservedRunningTime="2026-08-02 09:39:02.599736473 +0000 UTC m=+364.040626925" Aug 02 09:39:08.557777 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:08.557748 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-df7456b48-j4kkw" Aug 02 09:39:13.585162 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:13.585127 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5694fb54d4-trvzp" Aug 02 09:39:15.775184 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:15.775147 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz"] Aug 02 09:39:15.783782 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:15.783751 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" Aug 02 09:39:15.786183 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:15.786158 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 02 09:39:15.786911 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:15.786896 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 02 09:39:15.786982 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:15.786896 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 02 09:39:15.787045 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:15.787014 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kqtwm\"" Aug 02 09:39:15.787155 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:15.787139 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 02 09:39:15.792053 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:15.792013 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz"] Aug 02 09:39:15.935090 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:15.935049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ad7691-ec0a-483c-af2e-1828652793b3-tls-certs\") pod \"kube-auth-proxy-7569d6b956-9hnwz\" (UID: \"c9ad7691-ec0a-483c-af2e-1828652793b3\") " pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" Aug 02 09:39:15.935090 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:15.935092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9ad7691-ec0a-483c-af2e-1828652793b3-tmp\") pod \"kube-auth-proxy-7569d6b956-9hnwz\" (UID: \"c9ad7691-ec0a-483c-af2e-1828652793b3\") " pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" Aug 02 09:39:15.935317 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:15.935205 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznhq\" (UniqueName: \"kubernetes.io/projected/c9ad7691-ec0a-483c-af2e-1828652793b3-kube-api-access-qznhq\") pod \"kube-auth-proxy-7569d6b956-9hnwz\" (UID: \"c9ad7691-ec0a-483c-af2e-1828652793b3\") " pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" Aug 02 09:39:16.036491 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:16.036391 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ad7691-ec0a-483c-af2e-1828652793b3-tls-certs\") pod \"kube-auth-proxy-7569d6b956-9hnwz\" (UID: \"c9ad7691-ec0a-483c-af2e-1828652793b3\") " pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" Aug 02 09:39:16.036491 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:16.036425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9ad7691-ec0a-483c-af2e-1828652793b3-tmp\") pod \"kube-auth-proxy-7569d6b956-9hnwz\" (UID: \"c9ad7691-ec0a-483c-af2e-1828652793b3\") " pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" Aug 02 09:39:16.036642 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:16.036510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznhq\" (UniqueName: \"kubernetes.io/projected/c9ad7691-ec0a-483c-af2e-1828652793b3-kube-api-access-qznhq\") pod \"kube-auth-proxy-7569d6b956-9hnwz\" (UID: \"c9ad7691-ec0a-483c-af2e-1828652793b3\") " pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" Aug 02 09:39:16.038549 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:16.038523 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9ad7691-ec0a-483c-af2e-1828652793b3-tmp\") pod \"kube-auth-proxy-7569d6b956-9hnwz\" (UID: \"c9ad7691-ec0a-483c-af2e-1828652793b3\") " pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" Aug 02 09:39:16.038772 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:16.038751 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ad7691-ec0a-483c-af2e-1828652793b3-tls-certs\") pod \"kube-auth-proxy-7569d6b956-9hnwz\" (UID: \"c9ad7691-ec0a-483c-af2e-1828652793b3\") " pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" Aug 02 09:39:16.059969 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:16.059940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznhq\" (UniqueName: \"kubernetes.io/projected/c9ad7691-ec0a-483c-af2e-1828652793b3-kube-api-access-qznhq\") pod \"kube-auth-proxy-7569d6b956-9hnwz\" (UID: \"c9ad7691-ec0a-483c-af2e-1828652793b3\") " pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" Aug 02 09:39:16.095567 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:16.095541 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" Aug 02 09:39:16.230628 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:16.230528 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz"] Aug 02 09:39:16.233130 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:39:16.233100 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ad7691_ec0a_483c_af2e_1828652793b3.slice/crio-36bdfa692a57a246f1a2c1591b35d49bc5683bab996a00b0f81d755e8d3cbf22 WatchSource:0}: Error finding container 36bdfa692a57a246f1a2c1591b35d49bc5683bab996a00b0f81d755e8d3cbf22: Status 404 returned error can't find the container with id 36bdfa692a57a246f1a2c1591b35d49bc5683bab996a00b0f81d755e8d3cbf22 Aug 02 09:39:16.643131 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:16.643093 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" event={"ID":"c9ad7691-ec0a-483c-af2e-1828652793b3","Type":"ContainerStarted","Data":"36bdfa692a57a246f1a2c1591b35d49bc5683bab996a00b0f81d755e8d3cbf22"} Aug 02 09:39:19.659499 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:19.659453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" event={"ID":"c9ad7691-ec0a-483c-af2e-1828652793b3","Type":"ContainerStarted","Data":"1a8fc4dcae08d46cc18c4b2a1c418f1912819bca92691adaad637054a196b141"} Aug 02 09:39:19.675882 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:39:19.675829 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7569d6b956-9hnwz" podStartSLOduration=1.94183822 podStartE2EDuration="4.675814678s" totalImagesPullingTime="2.733976458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:39:19.67458446 +0000 UTC m=+381.115474918" watchObservedRunningTime="2026-08-02 09:39:19.675814678 +0000 UTC m=+381.116705181" Aug 02 09:40:39.747848 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:39.747813 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj"] Aug 02 09:40:39.751545 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:39.751523 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj" Aug 02 09:40:39.754953 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:39.754933 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-n7c9t\"" Aug 02 09:40:39.755044 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:39.755033 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 02 09:40:39.755088 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:39.755047 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Aug 02 09:40:39.755759 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:39.755746 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 02 09:40:39.762205 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:39.762184 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj"] Aug 02 09:40:39.851368 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:39.851338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvkzt\" (UniqueName: \"kubernetes.io/projected/3f02359d-42e9-41e0-8b82-80959b07b999-kube-api-access-lvkzt\") pod \"dns-operator-controller-manager-5985564776-lcbtj\" (UID: \"3f02359d-42e9-41e0-8b82-80959b07b999\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj" Aug 02 09:40:39.952124 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:39.952072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkzt\" (UniqueName: \"kubernetes.io/projected/3f02359d-42e9-41e0-8b82-80959b07b999-kube-api-access-lvkzt\") pod \"dns-operator-controller-manager-5985564776-lcbtj\" (UID: \"3f02359d-42e9-41e0-8b82-80959b07b999\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj" Aug 02 09:40:39.961996 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:39.961963 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkzt\" (UniqueName: \"kubernetes.io/projected/3f02359d-42e9-41e0-8b82-80959b07b999-kube-api-access-lvkzt\") pod \"dns-operator-controller-manager-5985564776-lcbtj\" (UID: \"3f02359d-42e9-41e0-8b82-80959b07b999\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj" Aug 02 09:40:40.062175 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:40.062119 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj" Aug 02 09:40:40.189272 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:40.189248 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj"] Aug 02 09:40:40.191570 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:40:40.191541 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f02359d_42e9_41e0_8b82_80959b07b999.slice/crio-5b4e4626acac1e0f4bce83a3319791663e6f5c7c258be53b0fba162a992dc68f WatchSource:0}: Error finding container 5b4e4626acac1e0f4bce83a3319791663e6f5c7c258be53b0fba162a992dc68f: Status 404 returned error can't find the container with id 5b4e4626acac1e0f4bce83a3319791663e6f5c7c258be53b0fba162a992dc68f Aug 02 09:40:41.015208 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:41.015165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj" event={"ID":"3f02359d-42e9-41e0-8b82-80959b07b999","Type":"ContainerStarted","Data":"5b4e4626acac1e0f4bce83a3319791663e6f5c7c258be53b0fba162a992dc68f"} Aug 02 09:40:44.466653 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.466621 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp"] Aug 02 09:40:44.470083 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.470066 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:44.474119 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.474093 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Aug 02 09:40:44.474228 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.474163 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rrmvk\"" Aug 02 09:40:44.474228 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.474106 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Aug 02 09:40:44.478671 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.478649 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp"] Aug 02 09:40:44.596978 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.596956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/189dbbf3-6076-40cb-8422-8259461886e3-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-nqhlp\" (UID: \"189dbbf3-6076-40cb-8422-8259461886e3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:44.597090 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.597001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/189dbbf3-6076-40cb-8422-8259461886e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-nqhlp\" (UID: \"189dbbf3-6076-40cb-8422-8259461886e3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:44.597131 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.597080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9h65\" (UniqueName: \"kubernetes.io/projected/189dbbf3-6076-40cb-8422-8259461886e3-kube-api-access-n9h65\") pod \"kuadrant-console-plugin-598f86468b-nqhlp\" (UID: \"189dbbf3-6076-40cb-8422-8259461886e3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:44.697660 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.697627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/189dbbf3-6076-40cb-8422-8259461886e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-nqhlp\" (UID: \"189dbbf3-6076-40cb-8422-8259461886e3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:44.697774 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.697691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9h65\" (UniqueName: \"kubernetes.io/projected/189dbbf3-6076-40cb-8422-8259461886e3-kube-api-access-n9h65\") pod \"kuadrant-console-plugin-598f86468b-nqhlp\" (UID: \"189dbbf3-6076-40cb-8422-8259461886e3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:44.697774 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.697754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/189dbbf3-6076-40cb-8422-8259461886e3-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-nqhlp\" (UID: \"189dbbf3-6076-40cb-8422-8259461886e3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:44.697774 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:40:44.697764 2564 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Aug 02 09:40:44.697944 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:40:44.697830 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/189dbbf3-6076-40cb-8422-8259461886e3-plugin-serving-cert podName:189dbbf3-6076-40cb-8422-8259461886e3 nodeName:}" failed. No retries permitted until 2026-08-02 09:40:45.197812921 +0000 UTC m=+466.638703355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/189dbbf3-6076-40cb-8422-8259461886e3-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-nqhlp" (UID: "189dbbf3-6076-40cb-8422-8259461886e3") : secret "plugin-serving-cert" not found Aug 02 09:40:44.698395 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.698373 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/189dbbf3-6076-40cb-8422-8259461886e3-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-nqhlp\" (UID: \"189dbbf3-6076-40cb-8422-8259461886e3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:44.706107 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:44.706079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9h65\" (UniqueName: \"kubernetes.io/projected/189dbbf3-6076-40cb-8422-8259461886e3-kube-api-access-n9h65\") pod \"kuadrant-console-plugin-598f86468b-nqhlp\" (UID: \"189dbbf3-6076-40cb-8422-8259461886e3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:45.034655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:45.034619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj" event={"ID":"3f02359d-42e9-41e0-8b82-80959b07b999","Type":"ContainerStarted","Data":"99dbf31ba3301cd41fb398f91b681a78d5f6a91887d63039fb8a0607338bbe28"} Aug 02 09:40:45.034838 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:45.034740 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj" Aug 02 09:40:45.051693 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:45.051646 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj" podStartSLOduration=2.192444736 podStartE2EDuration="6.05163442s" totalImagesPullingTime="3.859189684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:40:45.049686946 +0000 UTC m=+466.490577401" watchObservedRunningTime="2026-08-02 09:40:45.05163442 +0000 UTC m=+466.492524876" Aug 02 09:40:45.201519 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:45.201484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/189dbbf3-6076-40cb-8422-8259461886e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-nqhlp\" (UID: \"189dbbf3-6076-40cb-8422-8259461886e3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:45.203714 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:45.203694 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/189dbbf3-6076-40cb-8422-8259461886e3-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-nqhlp\" (UID: \"189dbbf3-6076-40cb-8422-8259461886e3\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:45.379617 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:45.379548 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" Aug 02 09:40:45.509645 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:45.509619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp"] Aug 02 09:40:45.512028 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:40:45.512000 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod189dbbf3_6076_40cb_8422_8259461886e3.slice/crio-8307de4f290c498152cc24c8035fd6964a9c0fc8bd8cc9bd9fe6f834f30e39e5 WatchSource:0}: Error finding container 8307de4f290c498152cc24c8035fd6964a9c0fc8bd8cc9bd9fe6f834f30e39e5: Status 404 returned error can't find the container with id 8307de4f290c498152cc24c8035fd6964a9c0fc8bd8cc9bd9fe6f834f30e39e5 Aug 02 09:40:46.040151 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:46.040112 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" event={"ID":"189dbbf3-6076-40cb-8422-8259461886e3","Type":"ContainerStarted","Data":"8307de4f290c498152cc24c8035fd6964a9c0fc8bd8cc9bd9fe6f834f30e39e5"} Aug 02 09:40:51.068266 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:51.068227 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" event={"ID":"189dbbf3-6076-40cb-8422-8259461886e3","Type":"ContainerStarted","Data":"ed75e24b28c19a6dd82a0969269f5325bbe9bcade3ad97f2df059c6cc9ed7725"} Aug 02 09:40:51.085009 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:51.084960 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-nqhlp" podStartSLOduration=2.407195262 podStartE2EDuration="7.084944714s" totalImagesPullingTime="4.677749452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:40:51.084355559 +0000 UTC m=+472.525246015" watchObservedRunningTime="2026-08-02 09:40:51.084944714 +0000 UTC m=+472.525835169" Aug 02 09:40:56.042221 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:40:56.042190 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-lcbtj" Aug 02 09:41:25.982740 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:25.982707 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:41:25.986092 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:25.986075 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" Aug 02 09:41:25.988573 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:25.988552 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 02 09:41:25.995303 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:25.995282 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:41:26.026756 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.026725 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:41:26.158508 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.158456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzvk2\" (UniqueName: \"kubernetes.io/projected/4047abd3-73ff-4253-8362-434614d61353-kube-api-access-wzvk2\") pod \"limitador-limitador-85d9db4d5d-7kqkt\" (UID: \"4047abd3-73ff-4253-8362-434614d61353\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" Aug 02 09:41:26.158606 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.158575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4047abd3-73ff-4253-8362-434614d61353-config-file\") pod \"limitador-limitador-85d9db4d5d-7kqkt\" (UID: \"4047abd3-73ff-4253-8362-434614d61353\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" Aug 02 09:41:26.259837 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.259776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzvk2\" (UniqueName: \"kubernetes.io/projected/4047abd3-73ff-4253-8362-434614d61353-kube-api-access-wzvk2\") pod \"limitador-limitador-85d9db4d5d-7kqkt\" (UID: \"4047abd3-73ff-4253-8362-434614d61353\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" Aug 02 09:41:26.259837 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.259830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4047abd3-73ff-4253-8362-434614d61353-config-file\") pod \"limitador-limitador-85d9db4d5d-7kqkt\" (UID: \"4047abd3-73ff-4253-8362-434614d61353\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" Aug 02 09:41:26.260384 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.260364 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4047abd3-73ff-4253-8362-434614d61353-config-file\") pod \"limitador-limitador-85d9db4d5d-7kqkt\" (UID: \"4047abd3-73ff-4253-8362-434614d61353\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" Aug 02 09:41:26.267562 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.267535 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzvk2\" (UniqueName: \"kubernetes.io/projected/4047abd3-73ff-4253-8362-434614d61353-kube-api-access-wzvk2\") pod \"limitador-limitador-85d9db4d5d-7kqkt\" (UID: \"4047abd3-73ff-4253-8362-434614d61353\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" Aug 02 09:41:26.297208 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.297187 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" Aug 02 09:41:26.432664 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.432640 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:41:26.434564 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:41:26.434536 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4047abd3_73ff_4253_8362_434614d61353.slice/crio-54a0d91a378fc227b8428386912d42da462776420b0b889798386e195ca2ca19 WatchSource:0}: Error finding container 54a0d91a378fc227b8428386912d42da462776420b0b889798386e195ca2ca19: Status 404 returned error can't find the container with id 54a0d91a378fc227b8428386912d42da462776420b0b889798386e195ca2ca19 Aug 02 09:41:26.546202 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.546140 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x8bsm"] Aug 02 09:41:26.550701 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.550684 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-x8bsm" Aug 02 09:41:26.553768 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.553741 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-844f9\"" Aug 02 09:41:26.555641 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.555618 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x8bsm"] Aug 02 09:41:26.663504 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.663450 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh48d\" (UniqueName: \"kubernetes.io/projected/a06247bf-5064-4653-bb50-2395403bdf10-kube-api-access-jh48d\") pod \"authorino-c548b5d7d-x8bsm\" (UID: \"a06247bf-5064-4653-bb50-2395403bdf10\") " pod="kuadrant-system/authorino-c548b5d7d-x8bsm" Aug 02 09:41:26.764350 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.764320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh48d\" (UniqueName: \"kubernetes.io/projected/a06247bf-5064-4653-bb50-2395403bdf10-kube-api-access-jh48d\") pod \"authorino-c548b5d7d-x8bsm\" (UID: \"a06247bf-5064-4653-bb50-2395403bdf10\") " pod="kuadrant-system/authorino-c548b5d7d-x8bsm" Aug 02 09:41:26.772734 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.772708 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh48d\" (UniqueName: \"kubernetes.io/projected/a06247bf-5064-4653-bb50-2395403bdf10-kube-api-access-jh48d\") pod \"authorino-c548b5d7d-x8bsm\" (UID: \"a06247bf-5064-4653-bb50-2395403bdf10\") " pod="kuadrant-system/authorino-c548b5d7d-x8bsm" Aug 02 09:41:26.860785 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.860725 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-x8bsm" Aug 02 09:41:26.987371 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:26.987346 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x8bsm"] Aug 02 09:41:26.989280 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:41:26.989253 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda06247bf_5064_4653_bb50_2395403bdf10.slice/crio-c7f14e6c98fc3fe2ba1e0299da149432b92bae6f087bf0b0cdf7938b7e931513 WatchSource:0}: Error finding container c7f14e6c98fc3fe2ba1e0299da149432b92bae6f087bf0b0cdf7938b7e931513: Status 404 returned error can't find the container with id c7f14e6c98fc3fe2ba1e0299da149432b92bae6f087bf0b0cdf7938b7e931513 Aug 02 09:41:27.238810 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:27.238775 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-x8bsm" event={"ID":"a06247bf-5064-4653-bb50-2395403bdf10","Type":"ContainerStarted","Data":"c7f14e6c98fc3fe2ba1e0299da149432b92bae6f087bf0b0cdf7938b7e931513"} Aug 02 09:41:27.239923 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:27.239899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" event={"ID":"4047abd3-73ff-4253-8362-434614d61353","Type":"ContainerStarted","Data":"54a0d91a378fc227b8428386912d42da462776420b0b889798386e195ca2ca19"} Aug 02 09:41:29.252856 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:29.252822 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-x8bsm" event={"ID":"a06247bf-5064-4653-bb50-2395403bdf10","Type":"ContainerStarted","Data":"a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2"} Aug 02 09:41:29.266849 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:29.266796 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-x8bsm" podStartSLOduration=1.354276621 podStartE2EDuration="3.266775775s" totalImagesPullingTime="1.912499154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:41:29.265753506 +0000 UTC m=+510.706643968" watchObservedRunningTime="2026-08-02 09:41:29.266775775 +0000 UTC m=+510.707666231" Aug 02 09:41:32.270296 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:32.270262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" event={"ID":"4047abd3-73ff-4253-8362-434614d61353","Type":"ContainerStarted","Data":"5caf456786de0f82e3f5063f3352b254f21d7cc72f779e1dfc36bc57cce1bf22"} Aug 02 09:41:32.270701 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:32.270371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" Aug 02 09:41:32.287117 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:32.287067 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" podStartSLOduration=1.597153018 podStartE2EDuration="7.287053992s" totalImagesPullingTime="5.689900974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:41:32.28557971 +0000 UTC m=+513.726470166" watchObservedRunningTime="2026-08-02 09:41:32.287053992 +0000 UTC m=+513.727944483" Aug 02 09:41:43.277174 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:43.277143 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt" Aug 02 09:41:59.705889 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:59.705816 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b9bfcfbcd-w8c7p"] Aug 02 09:41:59.709251 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:59.709232 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" Aug 02 09:41:59.711415 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:59.711395 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 02 09:41:59.715625 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:59.715603 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b9bfcfbcd-w8c7p"] Aug 02 09:41:59.837260 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:59.837229 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/093934e9-4e0a-40ac-b386-1968432474a9-tls-cert\") pod \"authorino-b9bfcfbcd-w8c7p\" (UID: \"093934e9-4e0a-40ac-b386-1968432474a9\") " pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" Aug 02 09:41:59.837402 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:59.837271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwwp\" (UniqueName: \"kubernetes.io/projected/093934e9-4e0a-40ac-b386-1968432474a9-kube-api-access-tfwwp\") pod \"authorino-b9bfcfbcd-w8c7p\" (UID: \"093934e9-4e0a-40ac-b386-1968432474a9\") " pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" Aug 02 09:41:59.938097 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:59.938061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/093934e9-4e0a-40ac-b386-1968432474a9-tls-cert\") pod \"authorino-b9bfcfbcd-w8c7p\" (UID: \"093934e9-4e0a-40ac-b386-1968432474a9\") " pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" Aug 02 09:41:59.938238 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:59.938105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwwp\" (UniqueName: \"kubernetes.io/projected/093934e9-4e0a-40ac-b386-1968432474a9-kube-api-access-tfwwp\") pod \"authorino-b9bfcfbcd-w8c7p\" (UID: \"093934e9-4e0a-40ac-b386-1968432474a9\") " pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" Aug 02 09:41:59.940397 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:59.940368 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/093934e9-4e0a-40ac-b386-1968432474a9-tls-cert\") pod \"authorino-b9bfcfbcd-w8c7p\" (UID: \"093934e9-4e0a-40ac-b386-1968432474a9\") " pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" Aug 02 09:41:59.945916 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:41:59.945888 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwwp\" (UniqueName: \"kubernetes.io/projected/093934e9-4e0a-40ac-b386-1968432474a9-kube-api-access-tfwwp\") pod \"authorino-b9bfcfbcd-w8c7p\" (UID: \"093934e9-4e0a-40ac-b386-1968432474a9\") " pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" Aug 02 09:42:00.018357 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:00.018304 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" Aug 02 09:42:00.146709 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:00.146681 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b9bfcfbcd-w8c7p"] Aug 02 09:42:00.149885 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:42:00.149854 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod093934e9_4e0a_40ac_b386_1968432474a9.slice/crio-e30742972ba41f1fcc4229024fdade79a6196d4cb1928af8d92871d8e71f4932 WatchSource:0}: Error finding container e30742972ba41f1fcc4229024fdade79a6196d4cb1928af8d92871d8e71f4932: Status 404 returned error can't find the container with id e30742972ba41f1fcc4229024fdade79a6196d4cb1928af8d92871d8e71f4932 Aug 02 09:42:00.398349 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:00.398274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" event={"ID":"093934e9-4e0a-40ac-b386-1968432474a9","Type":"ContainerStarted","Data":"e30742972ba41f1fcc4229024fdade79a6196d4cb1928af8d92871d8e71f4932"} Aug 02 09:42:01.405072 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:01.405031 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" event={"ID":"093934e9-4e0a-40ac-b386-1968432474a9","Type":"ContainerStarted","Data":"c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff"} Aug 02 09:42:01.420345 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:01.420298 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" podStartSLOduration=2.049101754 podStartE2EDuration="2.420284144s" totalImagesPullingTime="371.18239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:42:01.418915801 +0000 UTC m=+542.859806258" watchObservedRunningTime="2026-08-02 09:42:01.420284144 +0000 UTC m=+542.861174638" Aug 02 09:42:01.446522 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:01.446485 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x8bsm"] Aug 02 09:42:01.446696 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:01.446666 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-x8bsm" podUID="a06247bf-5064-4653-bb50-2395403bdf10" containerName="authorino" containerID="cri-o://a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2" gracePeriod=30 Aug 02 09:42:01.689721 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:01.689697 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-x8bsm" Aug 02 09:42:01.755091 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:01.755063 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh48d\" (UniqueName: \"kubernetes.io/projected/a06247bf-5064-4653-bb50-2395403bdf10-kube-api-access-jh48d\") pod \"a06247bf-5064-4653-bb50-2395403bdf10\" (UID: \"a06247bf-5064-4653-bb50-2395403bdf10\") " Aug 02 09:42:01.757105 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:01.757087 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06247bf-5064-4653-bb50-2395403bdf10-kube-api-access-jh48d" (OuterVolumeSpecName: "kube-api-access-jh48d") pod "a06247bf-5064-4653-bb50-2395403bdf10" (UID: "a06247bf-5064-4653-bb50-2395403bdf10"). InnerVolumeSpecName "kube-api-access-jh48d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 09:42:01.856007 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:01.855979 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh48d\" (UniqueName: \"kubernetes.io/projected/a06247bf-5064-4653-bb50-2395403bdf10-kube-api-access-jh48d\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:42:02.410406 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:02.410369 2564 generic.go:356] "Generic (PLEG): container finished" podID="a06247bf-5064-4653-bb50-2395403bdf10" containerID="a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2" exitCode=0 Aug 02 09:42:02.410892 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:02.410419 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-x8bsm" Aug 02 09:42:02.410892 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:02.410475 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-x8bsm" event={"ID":"a06247bf-5064-4653-bb50-2395403bdf10","Type":"ContainerDied","Data":"a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2"} Aug 02 09:42:02.410892 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:02.410524 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-x8bsm" event={"ID":"a06247bf-5064-4653-bb50-2395403bdf10","Type":"ContainerDied","Data":"c7f14e6c98fc3fe2ba1e0299da149432b92bae6f087bf0b0cdf7938b7e931513"} Aug 02 09:42:02.410892 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:02.410544 2564 scope.go:117] "RemoveContainer" containerID="a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2" Aug 02 09:42:02.419335 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:02.419319 2564 scope.go:117] "RemoveContainer" containerID="a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2" Aug 02 09:42:02.419606 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:42:02.419584 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2\": container with ID starting with a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2 not found: ID does not exist" containerID="a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2" Aug 02 09:42:02.419671 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:02.419614 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2"} err="failed to get container status \"a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2\": rpc error: code = NotFound desc = could not find container \"a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2\": container with ID starting with a89274ca1f5e54c2c36b1630d4e63bae98f63007daec2ab487097adb601d05a2 not found: ID does not exist" Aug 02 09:42:02.430049 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:02.430029 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x8bsm"] Aug 02 09:42:02.433351 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:02.433331 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x8bsm"] Aug 02 09:42:02.978837 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:02.978806 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a06247bf-5064-4653-bb50-2395403bdf10" path="/var/lib/kubelet/pods/a06247bf-5064-4653-bb50-2395403bdf10/volumes" Aug 02 09:42:42.983546 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:42.983510 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:42:58.159697 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:42:58.159662 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:43:04.203791 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:04.203754 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:43:08.622752 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:08.622716 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:43:22.750760 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:22.750682 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:43:49.143454 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.143426 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph"] Aug 02 09:43:49.143877 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.143820 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a06247bf-5064-4653-bb50-2395403bdf10" containerName="authorino" Aug 02 09:43:49.143877 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.143832 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a06247bf-5064-4653-bb50-2395403bdf10" containerName="authorino" Aug 02 09:43:49.143946 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.143914 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a06247bf-5064-4653-bb50-2395403bdf10" containerName="authorino" Aug 02 09:43:49.146706 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.146683 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.151804 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.151783 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jh287\"" Aug 02 09:43:49.152774 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.152751 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 02 09:43:49.152774 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.152772 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Aug 02 09:43:49.152954 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.152777 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 02 09:43:49.179717 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.179689 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph"] Aug 02 09:43:49.255978 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.255947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.256131 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.255985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.256131 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.256001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4shf\" (UniqueName: \"kubernetes.io/projected/870b531d-c265-4263-be18-ffc2a6d618f8-kube-api-access-b4shf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.256131 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.256098 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.256257 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.256157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.256257 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.256201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870b531d-c265-4263-be18-ffc2a6d618f8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.356538 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.356513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.356666 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.356544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870b531d-c265-4263-be18-ffc2a6d618f8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.356666 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.356592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.356666 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.356612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.356666 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.356628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4shf\" (UniqueName: \"kubernetes.io/projected/870b531d-c265-4263-be18-ffc2a6d618f8-kube-api-access-b4shf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.356666 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.356661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.357014 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.356952 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.357054 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.357021 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.357183 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.357158 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.358797 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.358769 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/870b531d-c265-4263-be18-ffc2a6d618f8-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.359075 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.359055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/870b531d-c265-4263-be18-ffc2a6d618f8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.372831 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.372807 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4shf\" (UniqueName: \"kubernetes.io/projected/870b531d-c265-4263-be18-ffc2a6d618f8-kube-api-access-b4shf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wq4ph\" (UID: \"870b531d-c265-4263-be18-ffc2a6d618f8\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.456927 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.456904 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:49.606905 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.606874 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph"] Aug 02 09:43:49.609967 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:43:49.609940 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870b531d_c265_4263_be18_ffc2a6d618f8.slice/crio-61c709a6c3c7b35ac25768c7b7424a3caea2b6fa2db2f9cbf4d30f529aa913df WatchSource:0}: Error finding container 61c709a6c3c7b35ac25768c7b7424a3caea2b6fa2db2f9cbf4d30f529aa913df: Status 404 returned error can't find the container with id 61c709a6c3c7b35ac25768c7b7424a3caea2b6fa2db2f9cbf4d30f529aa913df Aug 02 09:43:49.611316 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.611297 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 02 09:43:49.930284 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:49.930200 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" event={"ID":"870b531d-c265-4263-be18-ffc2a6d618f8","Type":"ContainerStarted","Data":"61c709a6c3c7b35ac25768c7b7424a3caea2b6fa2db2f9cbf4d30f529aa913df"} Aug 02 09:43:50.936262 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:50.936169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" event={"ID":"870b531d-c265-4263-be18-ffc2a6d618f8","Type":"ContainerStarted","Data":"41cfdb4c85444b0983b10632cfc389ba5e245e7175c63943e87fb1d6cad06215"} Aug 02 09:43:50.981855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:50.981803 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" podStartSLOduration=1.012009875 podStartE2EDuration="1.981786353s" totalImagesPullingTime="969.776478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:43:50.980184274 +0000 UTC m=+652.421074733" watchObservedRunningTime="2026-08-02 09:43:50.981786353 +0000 UTC m=+652.422676809" Aug 02 09:43:51.665625 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:51.665590 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:43:59.457915 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:59.457874 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:59.458386 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:59.457944 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:59.470213 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:59.470191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:43:59.991910 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:43:59.991872 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wq4ph" Aug 02 09:44:00.768388 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:00.768351 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:44:45.815132 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:45.815099 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65bc78c9d7-bl554"] Aug 02 09:44:45.818642 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:45.818624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65bc78c9d7-bl554" Aug 02 09:44:45.826047 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:45.826023 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65bc78c9d7-bl554"] Aug 02 09:44:45.950722 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:45.950685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92aa30b9-77c6-4b2d-8744-b7343c15e993-tls-cert\") pod \"authorino-65bc78c9d7-bl554\" (UID: \"92aa30b9-77c6-4b2d-8744-b7343c15e993\") " pod="kuadrant-system/authorino-65bc78c9d7-bl554" Aug 02 09:44:45.950896 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:45.950796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9rgc\" (UniqueName: \"kubernetes.io/projected/92aa30b9-77c6-4b2d-8744-b7343c15e993-kube-api-access-n9rgc\") pod \"authorino-65bc78c9d7-bl554\" (UID: \"92aa30b9-77c6-4b2d-8744-b7343c15e993\") " pod="kuadrant-system/authorino-65bc78c9d7-bl554" Aug 02 09:44:46.051814 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:46.051784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92aa30b9-77c6-4b2d-8744-b7343c15e993-tls-cert\") pod \"authorino-65bc78c9d7-bl554\" (UID: \"92aa30b9-77c6-4b2d-8744-b7343c15e993\") " pod="kuadrant-system/authorino-65bc78c9d7-bl554" Aug 02 09:44:46.051964 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:46.051860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9rgc\" (UniqueName: \"kubernetes.io/projected/92aa30b9-77c6-4b2d-8744-b7343c15e993-kube-api-access-n9rgc\") pod \"authorino-65bc78c9d7-bl554\" (UID: \"92aa30b9-77c6-4b2d-8744-b7343c15e993\") " pod="kuadrant-system/authorino-65bc78c9d7-bl554" Aug 02 09:44:46.054162 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:46.054134 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92aa30b9-77c6-4b2d-8744-b7343c15e993-tls-cert\") pod \"authorino-65bc78c9d7-bl554\" (UID: \"92aa30b9-77c6-4b2d-8744-b7343c15e993\") " pod="kuadrant-system/authorino-65bc78c9d7-bl554" Aug 02 09:44:46.060932 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:46.060909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9rgc\" (UniqueName: \"kubernetes.io/projected/92aa30b9-77c6-4b2d-8744-b7343c15e993-kube-api-access-n9rgc\") pod \"authorino-65bc78c9d7-bl554\" (UID: \"92aa30b9-77c6-4b2d-8744-b7343c15e993\") " pod="kuadrant-system/authorino-65bc78c9d7-bl554" Aug 02 09:44:46.128454 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:46.128406 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65bc78c9d7-bl554" Aug 02 09:44:46.253333 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:46.253304 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65bc78c9d7-bl554"] Aug 02 09:44:46.255091 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:44:46.255062 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92aa30b9_77c6_4b2d_8744_b7343c15e993.slice/crio-ceb6a990e9da3f889dc9567d1e9a07f075f0d3b4a09bd0e8dff403638f50c76e WatchSource:0}: Error finding container ceb6a990e9da3f889dc9567d1e9a07f075f0d3b4a09bd0e8dff403638f50c76e: Status 404 returned error can't find the container with id ceb6a990e9da3f889dc9567d1e9a07f075f0d3b4a09bd0e8dff403638f50c76e Aug 02 09:44:47.209550 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:47.209514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65bc78c9d7-bl554" event={"ID":"92aa30b9-77c6-4b2d-8744-b7343c15e993","Type":"ContainerStarted","Data":"ceb6a990e9da3f889dc9567d1e9a07f075f0d3b4a09bd0e8dff403638f50c76e"} Aug 02 09:44:48.217548 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:48.217494 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65bc78c9d7-bl554" event={"ID":"92aa30b9-77c6-4b2d-8744-b7343c15e993","Type":"ContainerStarted","Data":"ef811b1ebaffdbe64bfb3d2c2ca5e02e9695fb9727b096dba579806c5d423c7e"} Aug 02 09:44:48.233867 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:48.233815 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65bc78c9d7-bl554" podStartSLOduration=2.278440574 podStartE2EDuration="3.233798928s" totalImagesPullingTime="955.358354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:44:48.232375188 +0000 UTC m=+709.673265645" watchObservedRunningTime="2026-08-02 09:44:48.233798928 +0000 UTC m=+709.674689383" Aug 02 09:44:48.261492 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:48.260986 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b9bfcfbcd-w8c7p"] Aug 02 09:44:48.261492 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:48.261226 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" podUID="093934e9-4e0a-40ac-b386-1968432474a9" containerName="authorino" containerID="cri-o://c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff" gracePeriod=30 Aug 02 09:44:48.509655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:48.509628 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" Aug 02 09:44:48.676978 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:48.676938 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/093934e9-4e0a-40ac-b386-1968432474a9-tls-cert\") pod \"093934e9-4e0a-40ac-b386-1968432474a9\" (UID: \"093934e9-4e0a-40ac-b386-1968432474a9\") " Aug 02 09:44:48.677154 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:48.676989 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfwwp\" (UniqueName: \"kubernetes.io/projected/093934e9-4e0a-40ac-b386-1968432474a9-kube-api-access-tfwwp\") pod \"093934e9-4e0a-40ac-b386-1968432474a9\" (UID: \"093934e9-4e0a-40ac-b386-1968432474a9\") " Aug 02 09:44:48.678999 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:48.678955 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093934e9-4e0a-40ac-b386-1968432474a9-kube-api-access-tfwwp" (OuterVolumeSpecName: "kube-api-access-tfwwp") pod "093934e9-4e0a-40ac-b386-1968432474a9" (UID: "093934e9-4e0a-40ac-b386-1968432474a9"). InnerVolumeSpecName "kube-api-access-tfwwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 09:44:48.689350 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:48.689321 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093934e9-4e0a-40ac-b386-1968432474a9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "093934e9-4e0a-40ac-b386-1968432474a9" (UID: "093934e9-4e0a-40ac-b386-1968432474a9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:44:48.777963 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:48.777872 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/093934e9-4e0a-40ac-b386-1968432474a9-tls-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:44:48.777963 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:48.777909 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfwwp\" (UniqueName: \"kubernetes.io/projected/093934e9-4e0a-40ac-b386-1968432474a9-kube-api-access-tfwwp\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:44:49.224296 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:49.224263 2564 generic.go:356] "Generic (PLEG): container finished" podID="093934e9-4e0a-40ac-b386-1968432474a9" containerID="c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff" exitCode=0 Aug 02 09:44:49.224708 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:49.224307 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" Aug 02 09:44:49.224708 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:49.224341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" event={"ID":"093934e9-4e0a-40ac-b386-1968432474a9","Type":"ContainerDied","Data":"c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff"} Aug 02 09:44:49.224708 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:49.224381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b9bfcfbcd-w8c7p" event={"ID":"093934e9-4e0a-40ac-b386-1968432474a9","Type":"ContainerDied","Data":"e30742972ba41f1fcc4229024fdade79a6196d4cb1928af8d92871d8e71f4932"} Aug 02 09:44:49.224708 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:49.224398 2564 scope.go:117] "RemoveContainer" containerID="c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff" Aug 02 09:44:49.233144 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:49.233125 2564 scope.go:117] "RemoveContainer" containerID="c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff" Aug 02 09:44:49.233387 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:44:49.233365 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff\": container with ID starting with c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff not found: ID does not exist" containerID="c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff" Aug 02 09:44:49.233441 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:49.233395 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff"} err="failed to get container status \"c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff\": rpc error: code = NotFound desc = could not find container \"c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff\": container with ID starting with c1efae05144fd2d75f771a1297d898fb79f31c4f70c4b6541da3378603e3dfff not found: ID does not exist" Aug 02 09:44:49.241095 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:49.241074 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b9bfcfbcd-w8c7p"] Aug 02 09:44:49.248768 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:49.248749 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b9bfcfbcd-w8c7p"] Aug 02 09:44:50.979310 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:44:50.979276 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="093934e9-4e0a-40ac-b386-1968432474a9" path="/var/lib/kubelet/pods/093934e9-4e0a-40ac-b386-1968432474a9/volumes" Aug 02 09:45:17.587390 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:45:17.587348 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:45:27.386547 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:45:27.386512 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:45:38.892959 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:45:38.892923 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:47:18.987947 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:47:18.987911 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:47:28.084527 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:47:28.084493 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:48:24.887137 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:48:24.887094 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:48:32.091696 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:48:32.091658 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:48:47.782395 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:48:47.782359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:48:53.690524 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:48:53.690482 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:49:02.986573 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:49:02.986530 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:49:11.186313 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:49:11.186274 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:49:23.583293 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:49:23.583213 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:49:28.884605 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:49:28.884572 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:49:33.991885 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:49:33.991852 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:49:55.595853 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:49:55.595815 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:50:06.299095 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:50:06.299058 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:50:30.494913 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:50:30.494878 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:50:41.091119 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:50:41.091083 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:50:46.289177 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:50:46.289139 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:50:55.497488 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:50:55.497401 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:51:06.396336 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:51:06.396298 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:51:12.192962 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:51:12.192927 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:51:19.590103 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:51:19.590065 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:51:34.004857 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:51:34.004819 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:51:39.254915 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:51:39.254883 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:51:49.483657 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:51:49.483619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:51:54.490002 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:51:54.489968 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:52:01.801304 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:52:01.801268 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:52:09.995374 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:52:09.995338 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:52:14.986238 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:52:14.986208 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:52:25.791190 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:52:25.791113 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:54:40.590570 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:54:40.590534 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:54:45.508239 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:54:45.508202 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:55:10.298251 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:55:10.298219 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:55:25.188077 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:55:25.187993 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:55:31.198599 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:55:31.198562 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:55:40.989834 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:55:40.989802 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:55:46.593730 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:55:46.593689 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:55:56.387925 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:55:56.387888 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:56:01.397365 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:56:01.397330 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:56:11.691142 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:56:11.691107 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:56:16.685923 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:56:16.685890 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:56:27.082884 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:56:27.082852 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:56:33.792674 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:56:33.792638 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:56:44.086563 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:56:44.086528 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:57:07.086616 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:07.086524 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:57:14.887381 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:14.887342 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:57:15.998378 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:15.998340 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns"] Aug 02 09:57:15.998954 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:15.998926 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093934e9-4e0a-40ac-b386-1968432474a9" containerName="authorino" Aug 02 09:57:15.999027 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:15.998957 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="093934e9-4e0a-40ac-b386-1968432474a9" containerName="authorino" Aug 02 09:57:15.999136 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:15.999091 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="093934e9-4e0a-40ac-b386-1968432474a9" containerName="authorino" Aug 02 09:57:16.002714 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.002692 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.005345 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.005325 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-efab3de6-kserve-self-signed-certs\"" Aug 02 09:57:16.016662 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.016639 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns"] Aug 02 09:57:16.070676 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.070652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-dshm\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.070768 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.070679 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-home\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.070768 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.070702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-model-cache\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.070768 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.070720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3af7f457-869d-494b-b88b-e59d50f8c0b9-tls-certs\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.070889 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.070868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-tmp-dir\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.070926 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.070897 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jldw\" (UniqueName: \"kubernetes.io/projected/3af7f457-869d-494b-b88b-e59d50f8c0b9-kube-api-access-4jldw\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.171501 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.171450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-tmp-dir\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.171655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.171503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jldw\" (UniqueName: \"kubernetes.io/projected/3af7f457-869d-494b-b88b-e59d50f8c0b9-kube-api-access-4jldw\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.171655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.171559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-dshm\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.171655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.171575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-home\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.171655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.171597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-model-cache\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.171899 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.171871 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-tmp-dir\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.171899 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.171890 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-home\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.172027 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.171885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3af7f457-869d-494b-b88b-e59d50f8c0b9-tls-certs\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.172027 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.172012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-model-cache\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.173823 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.173801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-dshm\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.174085 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.174067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3af7f457-869d-494b-b88b-e59d50f8c0b9-tls-certs\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.179445 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.179426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jldw\" (UniqueName: \"kubernetes.io/projected/3af7f457-869d-494b-b88b-e59d50f8c0b9-kube-api-access-4jldw\") pod \"e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.312244 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.312190 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:16.442179 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.442156 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns"] Aug 02 09:57:16.444239 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:57:16.444199 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3af7f457_869d_494b_b88b_e59d50f8c0b9.slice/crio-78bd41c7d0706d69ebc3e9f5f858ca48d4af61022d9db3b92a9f012729c1289c WatchSource:0}: Error finding container 78bd41c7d0706d69ebc3e9f5f858ca48d4af61022d9db3b92a9f012729c1289c: Status 404 returned error can't find the container with id 78bd41c7d0706d69ebc3e9f5f858ca48d4af61022d9db3b92a9f012729c1289c Aug 02 09:57:16.797088 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.797051 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" event={"ID":"3af7f457-869d-494b-b88b-e59d50f8c0b9","Type":"ContainerStarted","Data":"896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870"} Aug 02 09:57:16.797278 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.797102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" event={"ID":"3af7f457-869d-494b-b88b-e59d50f8c0b9","Type":"ContainerStarted","Data":"78bd41c7d0706d69ebc3e9f5f858ca48d4af61022d9db3b92a9f012729c1289c"} Aug 02 09:57:16.813769 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:16.813717 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" podStartSLOduration=1.813702972 podStartE2EDuration="1.813702972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:57:16.811637081 +0000 UTC m=+1458.252527538" watchObservedRunningTime="2026-08-02 09:57:16.813702972 +0000 UTC m=+1458.254593428" Aug 02 09:57:24.691327 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:24.691291 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:57:26.313095 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:26.313018 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:26.313095 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:26.313063 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:26.325284 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:26.325259 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:26.856749 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:26.856716 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:57:29.207906 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.207876 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff"] Aug 02 09:57:29.211323 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.211295 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.213604 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.213585 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-efab3de6-kserve-self-signed-certs\"" Aug 02 09:57:29.223413 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.223391 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff"] Aug 02 09:57:29.288517 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.288488 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-tmp-dir\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.288655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.288529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-home\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.288655 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.288607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7102c5e-751c-4e87-8803-717d63b44d5e-tls-certs\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.288758 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.288724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-dshm\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.288807 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.288787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvhdz\" (UniqueName: \"kubernetes.io/projected/f7102c5e-751c-4e87-8803-717d63b44d5e-kube-api-access-cvhdz\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.288845 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.288829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-model-cache\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.389543 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.389514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-home\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.389658 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.389552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7102c5e-751c-4e87-8803-717d63b44d5e-tls-certs\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.389658 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.389608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-dshm\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.389750 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.389676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvhdz\" (UniqueName: \"kubernetes.io/projected/f7102c5e-751c-4e87-8803-717d63b44d5e-kube-api-access-cvhdz\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.389750 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.389716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-model-cache\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.389846 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.389747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-tmp-dir\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.389995 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.389973 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-home\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.390109 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.390014 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-tmp-dir\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.390109 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.390090 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-model-cache\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.391975 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.391945 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-dshm\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.392148 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.392133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7102c5e-751c-4e87-8803-717d63b44d5e-tls-certs\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.397542 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.397520 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvhdz\" (UniqueName: \"kubernetes.io/projected/f7102c5e-751c-4e87-8803-717d63b44d5e-kube-api-access-cvhdz\") pod \"e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.523447 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.523394 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:29.657347 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.657320 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff"] Aug 02 09:57:29.659243 ip-10-0-130-182 kubenswrapper[2564]: W0802 09:57:29.659212 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7102c5e_751c_4e87_8803_717d63b44d5e.slice/crio-6298a15dcccb7325c96c5b730698b42138a5438443093b630bda8e7cc6d96f65 WatchSource:0}: Error finding container 6298a15dcccb7325c96c5b730698b42138a5438443093b630bda8e7cc6d96f65: Status 404 returned error can't find the container with id 6298a15dcccb7325c96c5b730698b42138a5438443093b630bda8e7cc6d96f65 Aug 02 09:57:29.861687 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.861600 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" event={"ID":"f7102c5e-751c-4e87-8803-717d63b44d5e","Type":"ContainerStarted","Data":"3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4"} Aug 02 09:57:29.861687 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.861656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" event={"ID":"f7102c5e-751c-4e87-8803-717d63b44d5e","Type":"ContainerStarted","Data":"6298a15dcccb7325c96c5b730698b42138a5438443093b630bda8e7cc6d96f65"} Aug 02 09:57:29.877899 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:29.877851 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" podStartSLOduration=0.877834054 podStartE2EDuration="877.834054ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 09:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 09:57:29.876626411 +0000 UTC m=+1471.317516868" watchObservedRunningTime="2026-08-02 09:57:29.877834054 +0000 UTC m=+1471.318724509" Aug 02 09:57:31.091645 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:31.091610 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:57:39.524135 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:39.524105 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:39.524135 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:39.524144 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:39.536411 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:39.536389 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:39.919087 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:39.919006 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:57:50.101643 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:50.101611 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns"] Aug 02 09:57:51.671175 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:51.671138 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff"] Aug 02 09:57:51.889453 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:57:51.889415 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:58:03.589728 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:03.589693 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:58:05.120726 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.120667 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" podUID="3af7f457-869d-494b-b88b-e59d50f8c0b9" containerName="main" containerID="cri-o://896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870" gracePeriod=45 Aug 02 09:58:05.359817 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.359794 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:58:05.412820 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.412763 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3af7f457-869d-494b-b88b-e59d50f8c0b9-tls-certs\") pod \"3af7f457-869d-494b-b88b-e59d50f8c0b9\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " Aug 02 09:58:05.412933 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.412839 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-tmp-dir\") pod \"3af7f457-869d-494b-b88b-e59d50f8c0b9\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " Aug 02 09:58:05.412933 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.412882 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-home\") pod \"3af7f457-869d-494b-b88b-e59d50f8c0b9\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " Aug 02 09:58:05.412933 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.412919 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-model-cache\") pod \"3af7f457-869d-494b-b88b-e59d50f8c0b9\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " Aug 02 09:58:05.413044 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.412944 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jldw\" (UniqueName: \"kubernetes.io/projected/3af7f457-869d-494b-b88b-e59d50f8c0b9-kube-api-access-4jldw\") pod \"3af7f457-869d-494b-b88b-e59d50f8c0b9\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " Aug 02 09:58:05.413044 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.412970 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-dshm\") pod \"3af7f457-869d-494b-b88b-e59d50f8c0b9\" (UID: \"3af7f457-869d-494b-b88b-e59d50f8c0b9\") " Aug 02 09:58:05.413145 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.413119 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-home" (OuterVolumeSpecName: "home") pod "3af7f457-869d-494b-b88b-e59d50f8c0b9" (UID: "3af7f457-869d-494b-b88b-e59d50f8c0b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:58:05.413200 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.413129 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3af7f457-869d-494b-b88b-e59d50f8c0b9" (UID: "3af7f457-869d-494b-b88b-e59d50f8c0b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:58:05.413259 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.413202 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "3af7f457-869d-494b-b88b-e59d50f8c0b9" (UID: "3af7f457-869d-494b-b88b-e59d50f8c0b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:58:05.413314 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.413271 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:05.413314 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.413290 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:05.413314 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.413304 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:05.415001 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.414980 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af7f457-869d-494b-b88b-e59d50f8c0b9-kube-api-access-4jldw" (OuterVolumeSpecName: "kube-api-access-4jldw") pod "3af7f457-869d-494b-b88b-e59d50f8c0b9" (UID: "3af7f457-869d-494b-b88b-e59d50f8c0b9"). InnerVolumeSpecName "kube-api-access-4jldw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 09:58:05.415086 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.415017 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-dshm" (OuterVolumeSpecName: "dshm") pod "3af7f457-869d-494b-b88b-e59d50f8c0b9" (UID: "3af7f457-869d-494b-b88b-e59d50f8c0b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:58:05.415086 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.415019 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af7f457-869d-494b-b88b-e59d50f8c0b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3af7f457-869d-494b-b88b-e59d50f8c0b9" (UID: "3af7f457-869d-494b-b88b-e59d50f8c0b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:58:05.514669 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.514638 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3af7f457-869d-494b-b88b-e59d50f8c0b9-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:05.514669 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.514668 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jldw\" (UniqueName: \"kubernetes.io/projected/3af7f457-869d-494b-b88b-e59d50f8c0b9-kube-api-access-4jldw\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:05.514669 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:05.514678 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3af7f457-869d-494b-b88b-e59d50f8c0b9-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:06.035474 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.035430 2564 generic.go:356] "Generic (PLEG): container finished" podID="3af7f457-869d-494b-b88b-e59d50f8c0b9" containerID="896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870" exitCode=0 Aug 02 09:58:06.035637 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.035498 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" Aug 02 09:58:06.035637 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.035500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" event={"ID":"3af7f457-869d-494b-b88b-e59d50f8c0b9","Type":"ContainerDied","Data":"896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870"} Aug 02 09:58:06.035637 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.035544 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns" event={"ID":"3af7f457-869d-494b-b88b-e59d50f8c0b9","Type":"ContainerDied","Data":"78bd41c7d0706d69ebc3e9f5f858ca48d4af61022d9db3b92a9f012729c1289c"} Aug 02 09:58:06.035637 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.035562 2564 scope.go:117] "RemoveContainer" containerID="896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870" Aug 02 09:58:06.044397 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.044382 2564 scope.go:117] "RemoveContainer" containerID="896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870" Aug 02 09:58:06.044672 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:58:06.044654 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870\": container with ID starting with 896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870 not found: ID does not exist" containerID="896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870" Aug 02 09:58:06.044755 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.044683 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870"} err="failed to get container status \"896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870\": rpc error: code = NotFound desc = could not find container \"896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870\": container with ID starting with 896f629442bc60a6315ee6fff6d2eab5ade3fa78375cb20b4a64d8d8367e6870 not found: ID does not exist" Aug 02 09:58:06.058671 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.058644 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns"] Aug 02 09:58:06.060306 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.060287 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-efab3de6-kserve-5bfc5fc6ff-nrkns"] Aug 02 09:58:06.691236 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.691197 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" podUID="f7102c5e-751c-4e87-8803-717d63b44d5e" containerName="main" containerID="cri-o://3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4" gracePeriod=45 Aug 02 09:58:06.934842 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.934813 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:58:06.978525 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:06.978485 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af7f457-869d-494b-b88b-e59d50f8c0b9" path="/var/lib/kubelet/pods/3af7f457-869d-494b-b88b-e59d50f8c0b9/volumes" Aug 02 09:58:07.028422 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028399 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-model-cache\") pod \"f7102c5e-751c-4e87-8803-717d63b44d5e\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " Aug 02 09:58:07.028545 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028445 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-home\") pod \"f7102c5e-751c-4e87-8803-717d63b44d5e\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " Aug 02 09:58:07.028545 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028507 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7102c5e-751c-4e87-8803-717d63b44d5e-tls-certs\") pod \"f7102c5e-751c-4e87-8803-717d63b44d5e\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " Aug 02 09:58:07.028545 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028543 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-tmp-dir\") pod \"f7102c5e-751c-4e87-8803-717d63b44d5e\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " Aug 02 09:58:07.028704 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028561 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-dshm\") pod \"f7102c5e-751c-4e87-8803-717d63b44d5e\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " Aug 02 09:58:07.028704 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028619 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvhdz\" (UniqueName: \"kubernetes.io/projected/f7102c5e-751c-4e87-8803-717d63b44d5e-kube-api-access-cvhdz\") pod \"f7102c5e-751c-4e87-8803-717d63b44d5e\" (UID: \"f7102c5e-751c-4e87-8803-717d63b44d5e\") " Aug 02 09:58:07.028704 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028653 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7102c5e-751c-4e87-8803-717d63b44d5e" (UID: "f7102c5e-751c-4e87-8803-717d63b44d5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:58:07.028855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028766 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-home" (OuterVolumeSpecName: "home") pod "f7102c5e-751c-4e87-8803-717d63b44d5e" (UID: "f7102c5e-751c-4e87-8803-717d63b44d5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:58:07.028855 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028798 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7102c5e-751c-4e87-8803-717d63b44d5e" (UID: "f7102c5e-751c-4e87-8803-717d63b44d5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:58:07.028960 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028919 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:07.028960 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028937 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:07.028960 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.028950 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:07.030588 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.030560 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-dshm" (OuterVolumeSpecName: "dshm") pod "f7102c5e-751c-4e87-8803-717d63b44d5e" (UID: "f7102c5e-751c-4e87-8803-717d63b44d5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 09:58:07.030695 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.030605 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7102c5e-751c-4e87-8803-717d63b44d5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7102c5e-751c-4e87-8803-717d63b44d5e" (UID: "f7102c5e-751c-4e87-8803-717d63b44d5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 09:58:07.030695 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.030635 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7102c5e-751c-4e87-8803-717d63b44d5e-kube-api-access-cvhdz" (OuterVolumeSpecName: "kube-api-access-cvhdz") pod "f7102c5e-751c-4e87-8803-717d63b44d5e" (UID: "f7102c5e-751c-4e87-8803-717d63b44d5e"). InnerVolumeSpecName "kube-api-access-cvhdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 09:58:07.042148 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.042127 2564 generic.go:356] "Generic (PLEG): container finished" podID="f7102c5e-751c-4e87-8803-717d63b44d5e" containerID="3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4" exitCode=0 Aug 02 09:58:07.042240 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.042168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" event={"ID":"f7102c5e-751c-4e87-8803-717d63b44d5e","Type":"ContainerDied","Data":"3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4"} Aug 02 09:58:07.042240 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.042181 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" Aug 02 09:58:07.042240 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.042200 2564 scope.go:117] "RemoveContainer" containerID="3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4" Aug 02 09:58:07.042366 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.042190 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff" event={"ID":"f7102c5e-751c-4e87-8803-717d63b44d5e","Type":"ContainerDied","Data":"6298a15dcccb7325c96c5b730698b42138a5438443093b630bda8e7cc6d96f65"} Aug 02 09:58:07.051193 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.051178 2564 scope.go:117] "RemoveContainer" containerID="3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4" Aug 02 09:58:07.051450 ip-10-0-130-182 kubenswrapper[2564]: E0802 09:58:07.051433 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4\": container with ID starting with 3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4 not found: ID does not exist" containerID="3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4" Aug 02 09:58:07.051546 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.051532 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4"} err="failed to get container status \"3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4\": rpc error: code = NotFound desc = could not find container \"3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4\": container with ID starting with 3fe27c8ac930b67cc1f8b7ba145121400aa204712985c5814bbaa2daa9154ba4 not found: ID does not exist" Aug 02 09:58:07.064225 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.064205 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff"] Aug 02 09:58:07.067658 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.067639 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-efab3de6-kserve-57f48bb555-jp2ff"] Aug 02 09:58:07.130154 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.130133 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7102c5e-751c-4e87-8803-717d63b44d5e-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:07.130154 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.130153 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7102c5e-751c-4e87-8803-717d63b44d5e-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:07.130266 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:07.130162 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvhdz\" (UniqueName: \"kubernetes.io/projected/f7102c5e-751c-4e87-8803-717d63b44d5e-kube-api-access-cvhdz\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 09:58:08.978275 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:08.978243 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7102c5e-751c-4e87-8803-717d63b44d5e" path="/var/lib/kubelet/pods/f7102c5e-751c-4e87-8803-717d63b44d5e/volumes" Aug 02 09:58:11.394898 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:11.394865 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:58:23.985871 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:23.985831 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:58:31.194710 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:31.194597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:58:40.988277 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:40.988246 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:58:47.984703 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:47.984667 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:58:58.187443 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:58:58.187401 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:59:05.583679 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:59:05.583644 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:59:20.794374 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:59:20.794336 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:59:28.389225 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:59:28.389188 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:59:39.390901 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:59:39.390862 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:59:47.089198 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:59:47.089165 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 09:59:53.893352 ip-10-0-130-182 kubenswrapper[2564]: I0802 09:59:53.893268 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:00:00.086385 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:00:00.086350 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:00:06.492728 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:00:06.492692 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:00:15.600048 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:00:15.600014 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:00:22.293970 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:00:22.293931 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:00:33.487565 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:00:33.487530 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:00:40.790053 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:00:40.790019 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:00:50.581808 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:00:50.581768 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:01:05.781055 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:01:05.781019 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:01:11.681929 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:01:11.681898 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:01:18.193901 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:01:18.193857 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:01:28.891932 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:01:28.891834 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:01:37.986208 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:01:37.986172 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:01:43.393844 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:01:43.393807 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:01:58.688343 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:01:58.688310 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:02:19.623687 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.623651 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz"] Aug 02 10:02:19.624138 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.624055 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af7f457-869d-494b-b88b-e59d50f8c0b9" containerName="main" Aug 02 10:02:19.624138 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.624067 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af7f457-869d-494b-b88b-e59d50f8c0b9" containerName="main" Aug 02 10:02:19.624138 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.624081 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7102c5e-751c-4e87-8803-717d63b44d5e" containerName="main" Aug 02 10:02:19.624138 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.624091 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7102c5e-751c-4e87-8803-717d63b44d5e" containerName="main" Aug 02 10:02:19.624270 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.624157 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7102c5e-751c-4e87-8803-717d63b44d5e" containerName="main" Aug 02 10:02:19.624270 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.624169 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af7f457-869d-494b-b88b-e59d50f8c0b9" containerName="main" Aug 02 10:02:19.627273 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.627256 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz" Aug 02 10:02:19.629890 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.629868 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v7lxd\"" Aug 02 10:02:19.629990 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.629960 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 02 10:02:19.634824 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.634764 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz"] Aug 02 10:02:19.694175 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.694145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55b8495-84fd-4b50-89b8-1aaf0926069f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea933196-qxgmz\" (UID: \"c55b8495-84fd-4b50-89b8-1aaf0926069f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz" Aug 02 10:02:19.794956 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.794923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55b8495-84fd-4b50-89b8-1aaf0926069f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea933196-qxgmz\" (UID: \"c55b8495-84fd-4b50-89b8-1aaf0926069f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz" Aug 02 10:02:19.795575 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.795557 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55b8495-84fd-4b50-89b8-1aaf0926069f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea933196-qxgmz\" (UID: \"c55b8495-84fd-4b50-89b8-1aaf0926069f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz" Aug 02 10:02:19.939163 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:19.939087 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz" Aug 02 10:02:20.073117 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:20.073092 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz"] Aug 02 10:02:20.075447 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:02:20.075421 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55b8495_84fd_4b50_89b8_1aaf0926069f.slice/crio-e535f351423082bf6ea641922201b68f582fc9b76304c87e9ebceb143576f6e1 WatchSource:0}: Error finding container e535f351423082bf6ea641922201b68f582fc9b76304c87e9ebceb143576f6e1: Status 404 returned error can't find the container with id e535f351423082bf6ea641922201b68f582fc9b76304c87e9ebceb143576f6e1 Aug 02 10:02:20.077247 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:20.077231 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 02 10:02:20.237611 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:20.237579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz" event={"ID":"c55b8495-84fd-4b50-89b8-1aaf0926069f","Type":"ContainerStarted","Data":"e535f351423082bf6ea641922201b68f582fc9b76304c87e9ebceb143576f6e1"} Aug 02 10:02:23.255400 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:23.255365 2564 generic.go:356] "Generic (PLEG): container finished" podID="c55b8495-84fd-4b50-89b8-1aaf0926069f" containerID="fb4ebf297ff46d59079987ebb8e3221cc755b8984dd9454a0537a2edce10fbf2" exitCode=0 Aug 02 10:02:23.255833 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:23.255448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz" event={"ID":"c55b8495-84fd-4b50-89b8-1aaf0926069f","Type":"ContainerDied","Data":"fb4ebf297ff46d59079987ebb8e3221cc755b8984dd9454a0537a2edce10fbf2"} Aug 02 10:02:24.387181 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:24.387152 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz" Aug 02 10:02:24.434774 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:24.434741 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55b8495-84fd-4b50-89b8-1aaf0926069f-maas-api-service-ca\") pod \"c55b8495-84fd-4b50-89b8-1aaf0926069f\" (UID: \"c55b8495-84fd-4b50-89b8-1aaf0926069f\") " Aug 02 10:02:24.435047 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:24.435028 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55b8495-84fd-4b50-89b8-1aaf0926069f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c55b8495-84fd-4b50-89b8-1aaf0926069f" (UID: "c55b8495-84fd-4b50-89b8-1aaf0926069f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:02:24.535605 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:24.535516 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c55b8495-84fd-4b50-89b8-1aaf0926069f-maas-api-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:02:25.266507 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:25.266394 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz" event={"ID":"c55b8495-84fd-4b50-89b8-1aaf0926069f","Type":"ContainerDied","Data":"e535f351423082bf6ea641922201b68f582fc9b76304c87e9ebceb143576f6e1"} Aug 02 10:02:25.266507 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:25.266432 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz" Aug 02 10:02:25.266507 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:25.266446 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e535f351423082bf6ea641922201b68f582fc9b76304c87e9ebceb143576f6e1" Aug 02 10:02:27.673623 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:27.673590 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz"] Aug 02 10:02:27.677500 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:27.677473 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea933196-qxgmz"] Aug 02 10:02:28.788413 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.788381 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm"] Aug 02 10:02:28.788905 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.788763 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55b8495-84fd-4b50-89b8-1aaf0926069f" containerName="revoke-keys" Aug 02 10:02:28.788905 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.788775 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55b8495-84fd-4b50-89b8-1aaf0926069f" containerName="revoke-keys" Aug 02 10:02:28.788905 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.788853 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c55b8495-84fd-4b50-89b8-1aaf0926069f" containerName="revoke-keys" Aug 02 10:02:28.792154 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.792126 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.794431 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.794408 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 02 10:02:28.794581 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.794561 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4d0cd833-gw-openshift-default-dockercfg-2xhss\"" Aug 02 10:02:28.805711 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.805688 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm"] Aug 02 10:02:28.875619 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.875589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-certs\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.875727 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.875632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-token\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.875727 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.875655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-envoy\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.875727 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.875700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-credential-socket\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.875844 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.875756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-data\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.875844 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.875806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-socket\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.875844 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.875822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-podinfo\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.875844 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.875838 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgpmp\" (UniqueName: \"kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-kube-api-access-tgpmp\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.875997 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.875893 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295641e2-ccac-4b83-bcd6-eee4023dca5d-istiod-ca-cert\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.890192 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.890163 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:02:28.976269 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-data\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.976366 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-socket\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.976366 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976302 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-podinfo\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.976366 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgpmp\" (UniqueName: \"kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-kube-api-access-tgpmp\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.976366 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295641e2-ccac-4b83-bcd6-eee4023dca5d-istiod-ca-cert\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.976573 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-certs\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.976573 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-token\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.976573 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-envoy\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.976573 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-credential-socket\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.976773 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-data\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.976959 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976937 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-socket\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.977062 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.976982 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-credential-socket\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.977243 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.977215 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-certs\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.977349 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.977225 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295641e2-ccac-4b83-bcd6-eee4023dca5d-istiod-ca-cert\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.979290 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.979269 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-envoy\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.979395 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.979312 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c55b8495-84fd-4b50-89b8-1aaf0926069f" path="/var/lib/kubelet/pods/c55b8495-84fd-4b50-89b8-1aaf0926069f/volumes" Aug 02 10:02:28.979395 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.979382 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-podinfo\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.989098 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.989074 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-token\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:28.989382 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:28.989365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgpmp\" (UniqueName: \"kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-kube-api-access-tgpmp\") pod \"e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:29.102632 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:29.102567 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:29.252685 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:29.252659 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm"] Aug 02 10:02:29.255003 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:02:29.254975 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295641e2_ccac_4b83_bcd6_eee4023dca5d.slice/crio-5ccf13def9f2ac2216494d45c41b9c018e8f2818518f5cf3ed4cd5f1388dd6c4 WatchSource:0}: Error finding container 5ccf13def9f2ac2216494d45c41b9c018e8f2818518f5cf3ed4cd5f1388dd6c4: Status 404 returned error can't find the container with id 5ccf13def9f2ac2216494d45c41b9c018e8f2818518f5cf3ed4cd5f1388dd6c4 Aug 02 10:02:29.287452 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:29.287420 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" event={"ID":"295641e2-ccac-4b83-bcd6-eee4023dca5d","Type":"ContainerStarted","Data":"5ccf13def9f2ac2216494d45c41b9c018e8f2818518f5cf3ed4cd5f1388dd6c4"} Aug 02 10:02:31.811918 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:31.811877 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:02:31.812208 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:31.811953 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:02:31.812208 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:31.811981 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:02:32.303982 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:32.303939 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" event={"ID":"295641e2-ccac-4b83-bcd6-eee4023dca5d","Type":"ContainerStarted","Data":"e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0"} Aug 02 10:02:32.325776 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:32.325731 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" podStartSLOduration=1.771474874 podStartE2EDuration="4.325715972s" totalImagesPullingTime="2.554241098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:02:32.323244773 +0000 UTC m=+1773.764135229" watchObservedRunningTime="2026-08-02 10:02:32.325715972 +0000 UTC m=+1773.766606427" Aug 02 10:02:33.103668 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:33.103634 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:33.108145 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:33.108121 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:33.309391 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:33.309358 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:33.310282 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:33.310256 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:37.295864 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:37.295826 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:02:47.391038 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:47.390942 2564 generic.go:356] "Generic (PLEG): container finished" podID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerID="e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0" exitCode=0 Aug 02 10:02:47.391038 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:47.391015 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" event={"ID":"295641e2-ccac-4b83-bcd6-eee4023dca5d","Type":"ContainerDied","Data":"e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0"} Aug 02 10:02:47.391527 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:47.391436 2564 scope.go:117] "RemoveContainer" containerID="e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0" Aug 02 10:02:47.393626 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:47.393596 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:02:47.393706 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:47.393674 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:02:47.393743 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:47.393719 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:02:48.310090 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:48.310053 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:48.404033 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:48.403988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" event={"ID":"295641e2-ccac-4b83-bcd6-eee4023dca5d","Type":"ContainerStarted","Data":"d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8"} Aug 02 10:02:49.103251 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:49.103217 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:49.104392 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:49.104362 2564 patch_prober.go:28] interesting pod/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Aug 02 10:02:49.104530 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:49.104417 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Aug 02 10:02:50.103159 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:50.103127 2564 patch_prober.go:28] interesting pod/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Aug 02 10:02:50.103669 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:50.103181 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Aug 02 10:02:51.103879 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:51.103796 2564 patch_prober.go:28] interesting pod/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Aug 02 10:02:51.103879 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:51.103859 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Aug 02 10:02:52.103485 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:52.103435 2564 patch_prober.go:28] interesting pod/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Aug 02 10:02:52.103654 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:52.103505 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Aug 02 10:02:53.103761 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:53.103724 2564 patch_prober.go:28] interesting pod/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Aug 02 10:02:53.104153 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:53.103778 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Aug 02 10:02:54.107280 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:54.107252 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:54.107661 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:54.107567 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:02:54.108304 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:02:54.108287 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:03:15.652735 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:15.652698 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm"] Aug 02 10:03:15.653183 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:15.652908 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" containerID="cri-o://d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8" gracePeriod=30 Aug 02 10:03:16.442625 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.442591 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms"] Aug 02 10:03:16.446273 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.446251 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.448937 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.448911 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-973f165e-openshift-default-dockercfg-f7ntj\"" Aug 02 10:03:16.457681 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.457659 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms"] Aug 02 10:03:16.601276 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.601242 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istiod-ca-cert\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.601416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.601292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-token\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.601416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.601366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-socket\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.601574 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.601438 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-envoy\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.601574 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.601502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-credential-socket\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.601574 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.601552 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-certs\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.601785 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.601585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-data\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.601785 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.601677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-podinfo\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.601785 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.601768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzgjf\" (UniqueName: \"kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-kube-api-access-bzgjf\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.702898 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.702817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzgjf\" (UniqueName: \"kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-kube-api-access-bzgjf\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.702898 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.702854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istiod-ca-cert\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.702898 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.702875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-token\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.703447 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.702936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-socket\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.703447 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.702967 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-envoy\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.703447 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.703031 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-credential-socket\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.703447 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.703083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-certs\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.703447 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.703159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-data\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.703447 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.703243 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-podinfo\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.703447 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.703372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-socket\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.703447 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.703401 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-certs\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.703747 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.703528 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-credential-socket\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.703747 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.703600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-data\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.703747 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.703674 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istiod-ca-cert\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.705230 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.705211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-envoy\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.705598 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.705580 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-podinfo\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.710627 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.710601 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-token\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.710801 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.710781 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzgjf\" (UniqueName: \"kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-kube-api-access-bzgjf\") pod \"e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.757397 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.757370 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:16.890357 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.890331 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms"] Aug 02 10:03:16.892676 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:03:16.892649 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc219b53_c39d_4f19_bb4b_18e39e3a1bc9.slice/crio-de6a082562a445b6e01895e16c8cb7a31b8fc81a27766ebacdc9d9dc5a3aed63 WatchSource:0}: Error finding container de6a082562a445b6e01895e16c8cb7a31b8fc81a27766ebacdc9d9dc5a3aed63: Status 404 returned error can't find the container with id de6a082562a445b6e01895e16c8cb7a31b8fc81a27766ebacdc9d9dc5a3aed63 Aug 02 10:03:16.894961 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.894928 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:03:16.895038 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.894995 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:03:16.895038 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:16.895023 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:03:17.543857 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:17.543821 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" event={"ID":"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9","Type":"ContainerStarted","Data":"917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2"} Aug 02 10:03:17.544028 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:17.543869 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" event={"ID":"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9","Type":"ContainerStarted","Data":"de6a082562a445b6e01895e16c8cb7a31b8fc81a27766ebacdc9d9dc5a3aed63"} Aug 02 10:03:17.562315 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:17.562265 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" podStartSLOduration=1.562247969 podStartE2EDuration="1.562247969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:03:17.561363399 +0000 UTC m=+1819.002253857" watchObservedRunningTime="2026-08-02 10:03:17.562247969 +0000 UTC m=+1819.003138424" Aug 02 10:03:17.757947 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:17.757911 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:17.763171 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:17.763146 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:18.293327 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:18.293281 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:03:18.549755 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:18.549679 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:18.550526 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:18.550504 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:03:20.908349 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:20.908326 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:03:21.047007 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.046969 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-podinfo\") pod \"295641e2-ccac-4b83-bcd6-eee4023dca5d\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " Aug 02 10:03:21.047177 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047055 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgpmp\" (UniqueName: \"kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-kube-api-access-tgpmp\") pod \"295641e2-ccac-4b83-bcd6-eee4023dca5d\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " Aug 02 10:03:21.047177 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047080 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-socket\") pod \"295641e2-ccac-4b83-bcd6-eee4023dca5d\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " Aug 02 10:03:21.047177 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047105 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295641e2-ccac-4b83-bcd6-eee4023dca5d-istiod-ca-cert\") pod \"295641e2-ccac-4b83-bcd6-eee4023dca5d\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " Aug 02 10:03:21.047177 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047135 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-token\") pod \"295641e2-ccac-4b83-bcd6-eee4023dca5d\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " Aug 02 10:03:21.047395 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047219 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-envoy\") pod \"295641e2-ccac-4b83-bcd6-eee4023dca5d\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " Aug 02 10:03:21.047395 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047328 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-data\") pod \"295641e2-ccac-4b83-bcd6-eee4023dca5d\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " Aug 02 10:03:21.047395 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047363 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-credential-socket\") pod \"295641e2-ccac-4b83-bcd6-eee4023dca5d\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " Aug 02 10:03:21.047395 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047393 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-certs\") pod \"295641e2-ccac-4b83-bcd6-eee4023dca5d\" (UID: \"295641e2-ccac-4b83-bcd6-eee4023dca5d\") " Aug 02 10:03:21.047639 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047405 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "295641e2-ccac-4b83-bcd6-eee4023dca5d" (UID: "295641e2-ccac-4b83-bcd6-eee4023dca5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:03:21.047639 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047612 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "295641e2-ccac-4b83-bcd6-eee4023dca5d" (UID: "295641e2-ccac-4b83-bcd6-eee4023dca5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:03:21.047743 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047637 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295641e2-ccac-4b83-bcd6-eee4023dca5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "295641e2-ccac-4b83-bcd6-eee4023dca5d" (UID: "295641e2-ccac-4b83-bcd6-eee4023dca5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:03:21.047743 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047652 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "295641e2-ccac-4b83-bcd6-eee4023dca5d" (UID: "295641e2-ccac-4b83-bcd6-eee4023dca5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:03:21.047868 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.047848 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "295641e2-ccac-4b83-bcd6-eee4023dca5d" (UID: "295641e2-ccac-4b83-bcd6-eee4023dca5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:03:21.048115 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.048093 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:03:21.048199 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.048119 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295641e2-ccac-4b83-bcd6-eee4023dca5d-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:03:21.048199 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.048131 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:03:21.048199 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.048143 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:03:21.049368 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.049345 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "295641e2-ccac-4b83-bcd6-eee4023dca5d" (UID: "295641e2-ccac-4b83-bcd6-eee4023dca5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:03:21.049678 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.049658 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "295641e2-ccac-4b83-bcd6-eee4023dca5d" (UID: "295641e2-ccac-4b83-bcd6-eee4023dca5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:03:21.049678 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.049664 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-kube-api-access-tgpmp" (OuterVolumeSpecName: "kube-api-access-tgpmp") pod "295641e2-ccac-4b83-bcd6-eee4023dca5d" (UID: "295641e2-ccac-4b83-bcd6-eee4023dca5d"). InnerVolumeSpecName "kube-api-access-tgpmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:03:21.049804 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.049733 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "295641e2-ccac-4b83-bcd6-eee4023dca5d" (UID: "295641e2-ccac-4b83-bcd6-eee4023dca5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:03:21.149469 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.149446 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgpmp\" (UniqueName: \"kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-kube-api-access-tgpmp\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:03:21.149597 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.149478 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:03:21.149597 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.149489 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:03:21.149597 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.149498 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295641e2-ccac-4b83-bcd6-eee4023dca5d-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:03:21.149597 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.149505 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295641e2-ccac-4b83-bcd6-eee4023dca5d-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:03:21.566951 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.566913 2564 generic.go:356] "Generic (PLEG): container finished" podID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerID="d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8" exitCode=0 Aug 02 10:03:21.567180 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.566946 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" event={"ID":"295641e2-ccac-4b83-bcd6-eee4023dca5d","Type":"ContainerDied","Data":"d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8"} Aug 02 10:03:21.567180 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.566965 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" Aug 02 10:03:21.567180 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.567004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm" event={"ID":"295641e2-ccac-4b83-bcd6-eee4023dca5d","Type":"ContainerDied","Data":"5ccf13def9f2ac2216494d45c41b9c018e8f2818518f5cf3ed4cd5f1388dd6c4"} Aug 02 10:03:21.567180 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.567018 2564 scope.go:117] "RemoveContainer" containerID="d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8" Aug 02 10:03:21.576123 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.576107 2564 scope.go:117] "RemoveContainer" containerID="e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0" Aug 02 10:03:21.584599 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.584583 2564 scope.go:117] "RemoveContainer" containerID="d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8" Aug 02 10:03:21.584820 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:03:21.584801 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8\": container with ID starting with d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8 not found: ID does not exist" containerID="d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8" Aug 02 10:03:21.584881 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.584828 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8"} err="failed to get container status \"d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8\": rpc error: code = NotFound desc = could not find container \"d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8\": container with ID starting with d128f47dc53db1136b2c6918061c141d8f3ff101cbd805fabf3a76e73eb8f6e8 not found: ID does not exist" Aug 02 10:03:21.584881 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.584851 2564 scope.go:117] "RemoveContainer" containerID="e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0" Aug 02 10:03:21.585078 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:03:21.585054 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0\": container with ID starting with e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0 not found: ID does not exist" containerID="e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0" Aug 02 10:03:21.585120 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.585089 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0"} err="failed to get container status \"e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0\": rpc error: code = NotFound desc = could not find container \"e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0\": container with ID starting with e1ec69d2fe7f605342b0a46788b2b54d902fff1442ef01145c93b8496453e0b0 not found: ID does not exist" Aug 02 10:03:21.587933 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.587912 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm"] Aug 02 10:03:21.591980 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:21.591959 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4d0cd833-gw-openshift-default-578bdc688d-xtvrm"] Aug 02 10:03:22.981368 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:22.981334 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" path="/var/lib/kubelet/pods/295641e2-ccac-4b83-bcd6-eee4023dca5d/volumes" Aug 02 10:03:26.487199 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:26.487164 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:03:52.057842 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.057798 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms"] Aug 02 10:03:52.058776 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.058722 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" podUID="dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" containerName="istio-proxy" containerID="cri-o://917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2" gracePeriod=30 Aug 02 10:03:52.788226 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.788178 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f"] Aug 02 10:03:52.788729 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.788710 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" Aug 02 10:03:52.788729 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.788729 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" Aug 02 10:03:52.788951 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.788937 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" Aug 02 10:03:52.789017 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.788952 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" Aug 02 10:03:52.792249 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.792228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:52.794593 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.794575 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4ad08e62-openshift-default-dockercfg-tbm47\"" Aug 02 10:03:52.803265 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.803243 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f"] Aug 02 10:03:52.922760 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.922730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istiod-ca-cert\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:52.922957 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.922782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-socket\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:52.922957 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.922865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-data\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:52.922957 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.922949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-podinfo\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:52.923105 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.922983 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-token\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:52.923105 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.923044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-envoy\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:52.923105 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.923073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-certs\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:52.923105 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.923093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-credential-socket\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:52.923253 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:52.923118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5lr4\" (UniqueName: \"kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-kube-api-access-t5lr4\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.024056 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istiod-ca-cert\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.024246 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-socket\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.024246 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-data\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.024246 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-podinfo\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.024246 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-token\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.024499 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024272 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-envoy\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.024499 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-certs\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.024499 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-credential-socket\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.024499 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5lr4\" (UniqueName: \"kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-kube-api-access-t5lr4\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.024718 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024514 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-data\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.024718 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024604 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-socket\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.025060 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.024841 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-credential-socket\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.025225 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.025199 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istiod-ca-cert\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.025447 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.025427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-certs\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.027122 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.027094 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-podinfo\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.027269 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.027252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-envoy\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.031810 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.031783 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-token\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.031898 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.031788 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5lr4\" (UniqueName: \"kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-kube-api-access-t5lr4\") pod \"e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.102791 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.102706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:53.285630 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.285597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f"] Aug 02 10:03:53.288228 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:03:53.288195 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163b4b62_b2c8_4147_9e62_e0dfda89d7a2.slice/crio-84551ec101dc2e00b8c4af23b9009ee3cc69aee55c32822038596526e7239529 WatchSource:0}: Error finding container 84551ec101dc2e00b8c4af23b9009ee3cc69aee55c32822038596526e7239529: Status 404 returned error can't find the container with id 84551ec101dc2e00b8c4af23b9009ee3cc69aee55c32822038596526e7239529 Aug 02 10:03:53.293607 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.293573 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:03:53.293680 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.293637 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:03:53.293680 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.293665 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:03:53.693850 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.693768 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:03:53.726088 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.726048 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" event={"ID":"163b4b62-b2c8-4147-9e62-e0dfda89d7a2","Type":"ContainerStarted","Data":"858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62"} Aug 02 10:03:53.726264 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.726111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" event={"ID":"163b4b62-b2c8-4147-9e62-e0dfda89d7a2","Type":"ContainerStarted","Data":"84551ec101dc2e00b8c4af23b9009ee3cc69aee55c32822038596526e7239529"} Aug 02 10:03:53.752277 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:53.749755 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" podStartSLOduration=1.74973479 podStartE2EDuration="1.74973479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:03:53.749232447 +0000 UTC m=+1855.190122903" watchObservedRunningTime="2026-08-02 10:03:53.74973479 +0000 UTC m=+1855.190625247" Aug 02 10:03:54.103576 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:54.103534 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:54.108933 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:54.108908 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:54.731201 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:54.731165 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:03:54.732317 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:03:54.732299 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:04:00.020616 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.020590 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:04:00.115972 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.115892 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istiod-ca-cert\") pod \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " Aug 02 10:04:00.115972 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.115946 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-certs\") pod \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " Aug 02 10:04:00.116174 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.115977 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-podinfo\") pod \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " Aug 02 10:04:00.116174 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116003 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-token\") pod \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " Aug 02 10:04:00.116174 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116024 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-envoy\") pod \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " Aug 02 10:04:00.116174 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116055 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-data\") pod \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " Aug 02 10:04:00.116174 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116103 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzgjf\" (UniqueName: \"kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-kube-api-access-bzgjf\") pod \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " Aug 02 10:04:00.116174 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116127 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-socket\") pod \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " Aug 02 10:04:00.116174 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116172 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-credential-socket\") pod \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\" (UID: \"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9\") " Aug 02 10:04:00.116581 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116240 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" (UID: "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:04:00.116581 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116325 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" (UID: "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:04:00.116581 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116549 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" (UID: "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:04:00.116744 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116640 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" (UID: "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:04:00.116744 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116557 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" (UID: "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:04:00.116744 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116660 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:00.116744 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116681 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:00.116744 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.116695 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:00.118194 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.118168 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" (UID: "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:04:00.118332 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.118297 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-kube-api-access-bzgjf" (OuterVolumeSpecName: "kube-api-access-bzgjf") pod "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" (UID: "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9"). InnerVolumeSpecName "kube-api-access-bzgjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:04:00.118490 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.118452 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" (UID: "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:04:00.118575 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.118490 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" (UID: "dc219b53-c39d-4f19-bb4b-18e39e3a1bc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:04:00.217178 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.217143 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:00.217178 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.217175 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:00.217178 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.217184 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:00.217401 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.217192 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:00.217401 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.217201 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzgjf\" (UniqueName: \"kubernetes.io/projected/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-kube-api-access-bzgjf\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:00.217401 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.217211 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:00.771579 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.771549 2564 generic.go:356] "Generic (PLEG): container finished" podID="dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" containerID="917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2" exitCode=0 Aug 02 10:04:00.771795 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.771586 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" event={"ID":"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9","Type":"ContainerDied","Data":"917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2"} Aug 02 10:04:00.771795 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.771606 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" Aug 02 10:04:00.771795 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.771651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms" event={"ID":"dc219b53-c39d-4f19-bb4b-18e39e3a1bc9","Type":"ContainerDied","Data":"de6a082562a445b6e01895e16c8cb7a31b8fc81a27766ebacdc9d9dc5a3aed63"} Aug 02 10:04:00.771795 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.771673 2564 scope.go:117] "RemoveContainer" containerID="917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2" Aug 02 10:04:00.781204 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.781186 2564 scope.go:117] "RemoveContainer" containerID="917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2" Aug 02 10:04:00.781449 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:04:00.781429 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2\": container with ID starting with 917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2 not found: ID does not exist" containerID="917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2" Aug 02 10:04:00.781522 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.781458 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2"} err="failed to get container status \"917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2\": rpc error: code = NotFound desc = could not find container \"917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2\": container with ID starting with 917dc772669fa489c101e83afae0597482b3e4ccf3f840a268dc680b5ced45c2 not found: ID does not exist" Aug 02 10:04:00.792198 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.792177 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms"] Aug 02 10:04:00.798166 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.798146 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-973f165e-openshift-default-79db84c5f6-5mcms"] Aug 02 10:04:00.979014 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:00.978978 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" path="/var/lib/kubelet/pods/dc219b53-c39d-4f19-bb4b-18e39e3a1bc9/volumes" Aug 02 10:04:01.894701 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:01.894667 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:04:07.815073 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:07.815038 2564 generic.go:356] "Generic (PLEG): container finished" podID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerID="858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62" exitCode=0 Aug 02 10:04:07.815494 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:07.815109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" event={"ID":"163b4b62-b2c8-4147-9e62-e0dfda89d7a2","Type":"ContainerDied","Data":"858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62"} Aug 02 10:04:07.815549 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:07.815498 2564 scope.go:117] "RemoveContainer" containerID="858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62" Aug 02 10:04:07.816608 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:07.816578 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:04:07.816679 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:07.816661 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:04:07.816723 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:07.816708 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:04:08.822494 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:08.822443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" event={"ID":"163b4b62-b2c8-4147-9e62-e0dfda89d7a2","Type":"ContainerStarted","Data":"3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8"} Aug 02 10:04:09.103839 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:09.103755 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:04:09.104651 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:09.104623 2564 patch_prober.go:28] interesting pod/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Aug 02 10:04:09.104780 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:09.104680 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Aug 02 10:04:09.731662 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:09.731622 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:04:10.103489 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:10.103392 2564 patch_prober.go:28] interesting pod/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Aug 02 10:04:10.103489 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:10.103447 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Aug 02 10:04:11.103633 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:11.103592 2564 patch_prober.go:28] interesting pod/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Aug 02 10:04:11.104085 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:11.103651 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Aug 02 10:04:12.104098 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:12.104063 2564 patch_prober.go:28] interesting pod/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Aug 02 10:04:12.104505 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:12.104130 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Aug 02 10:04:13.103763 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:13.103725 2564 patch_prober.go:28] interesting pod/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Aug 02 10:04:13.103953 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:13.103814 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Aug 02 10:04:14.107167 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:14.107129 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:04:14.108073 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:14.108051 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:04:26.963935 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:26.963839 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f"] Aug 02 10:04:26.964418 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:26.964134 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" containerID="cri-o://3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8" gracePeriod=30 Aug 02 10:04:29.109502 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:29.109441 2564 patch_prober.go:28] interesting pod/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 02 10:04:29.109866 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:29.109522 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 02 10:04:29.695230 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:29.695194 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:04:32.210359 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.210334 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:04:32.320640 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.320558 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-socket\") pod \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " Aug 02 10:04:32.320640 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.320617 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istiod-ca-cert\") pod \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " Aug 02 10:04:32.320854 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.320652 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-envoy\") pod \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " Aug 02 10:04:32.320854 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.320676 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-data\") pod \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " Aug 02 10:04:32.320854 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.320696 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-token\") pod \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " Aug 02 10:04:32.320854 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.320732 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5lr4\" (UniqueName: \"kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-kube-api-access-t5lr4\") pod \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " Aug 02 10:04:32.320854 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.320788 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-certs\") pod \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " Aug 02 10:04:32.320854 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.320817 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-credential-socket\") pod \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " Aug 02 10:04:32.320854 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.320820 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "163b4b62-b2c8-4147-9e62-e0dfda89d7a2" (UID: "163b4b62-b2c8-4147-9e62-e0dfda89d7a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:04:32.320854 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.320854 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-podinfo\") pod \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\" (UID: \"163b4b62-b2c8-4147-9e62-e0dfda89d7a2\") " Aug 02 10:04:32.321254 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.321126 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "163b4b62-b2c8-4147-9e62-e0dfda89d7a2" (UID: "163b4b62-b2c8-4147-9e62-e0dfda89d7a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:04:32.321254 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.321122 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "163b4b62-b2c8-4147-9e62-e0dfda89d7a2" (UID: "163b4b62-b2c8-4147-9e62-e0dfda89d7a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:04:32.321254 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.321136 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "163b4b62-b2c8-4147-9e62-e0dfda89d7a2" (UID: "163b4b62-b2c8-4147-9e62-e0dfda89d7a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:04:32.321254 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.321187 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-data" (OuterVolumeSpecName: "istio-data") pod "163b4b62-b2c8-4147-9e62-e0dfda89d7a2" (UID: "163b4b62-b2c8-4147-9e62-e0dfda89d7a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:04:32.321254 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.321197 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:32.323013 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.322952 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "163b4b62-b2c8-4147-9e62-e0dfda89d7a2" (UID: "163b4b62-b2c8-4147-9e62-e0dfda89d7a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:04:32.323013 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.322982 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "163b4b62-b2c8-4147-9e62-e0dfda89d7a2" (UID: "163b4b62-b2c8-4147-9e62-e0dfda89d7a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:04:32.323151 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.323029 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-kube-api-access-t5lr4" (OuterVolumeSpecName: "kube-api-access-t5lr4") pod "163b4b62-b2c8-4147-9e62-e0dfda89d7a2" (UID: "163b4b62-b2c8-4147-9e62-e0dfda89d7a2"). InnerVolumeSpecName "kube-api-access-t5lr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:04:32.323151 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.323044 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "163b4b62-b2c8-4147-9e62-e0dfda89d7a2" (UID: "163b4b62-b2c8-4147-9e62-e0dfda89d7a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:04:32.422445 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.422414 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:32.422445 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.422442 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:32.422632 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.422454 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:32.422632 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.422482 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:32.422632 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.422494 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5lr4\" (UniqueName: \"kubernetes.io/projected/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-kube-api-access-t5lr4\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:32.422632 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.422507 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:32.422632 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.422520 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:32.422632 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.422531 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/163b4b62-b2c8-4147-9e62-e0dfda89d7a2-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:04:32.952753 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.952720 2564 generic.go:356] "Generic (PLEG): container finished" podID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerID="3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8" exitCode=0 Aug 02 10:04:32.952917 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.952765 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" event={"ID":"163b4b62-b2c8-4147-9e62-e0dfda89d7a2","Type":"ContainerDied","Data":"3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8"} Aug 02 10:04:32.952917 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.952804 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" event={"ID":"163b4b62-b2c8-4147-9e62-e0dfda89d7a2","Type":"ContainerDied","Data":"84551ec101dc2e00b8c4af23b9009ee3cc69aee55c32822038596526e7239529"} Aug 02 10:04:32.952917 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.952827 2564 scope.go:117] "RemoveContainer" containerID="3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8" Aug 02 10:04:32.952917 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.952766 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f" Aug 02 10:04:32.962289 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.962273 2564 scope.go:117] "RemoveContainer" containerID="858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62" Aug 02 10:04:32.969954 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.969937 2564 scope.go:117] "RemoveContainer" containerID="3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8" Aug 02 10:04:32.970201 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:04:32.970181 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8\": container with ID starting with 3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8 not found: ID does not exist" containerID="3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8" Aug 02 10:04:32.970274 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.970215 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8"} err="failed to get container status \"3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8\": rpc error: code = NotFound desc = could not find container \"3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8\": container with ID starting with 3e42f30a52a73d7deaaa80caab5f2cb20430799bb2a691191a0d8236c01a53c8 not found: ID does not exist" Aug 02 10:04:32.970274 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.970242 2564 scope.go:117] "RemoveContainer" containerID="858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62" Aug 02 10:04:32.970554 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:04:32.970529 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62\": container with ID starting with 858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62 not found: ID does not exist" containerID="858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62" Aug 02 10:04:32.970592 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.970564 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62"} err="failed to get container status \"858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62\": rpc error: code = NotFound desc = could not find container \"858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62\": container with ID starting with 858f131e443edc7a611b7a829e24e0943eea38fe3ad979b4981d50aaf098be62 not found: ID does not exist" Aug 02 10:04:32.979210 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.979187 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f"] Aug 02 10:04:32.979299 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:32.979222 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4ad08e62-openshift-default-5d875d9b68-dbd8f"] Aug 02 10:04:34.978084 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:34.978051 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" path="/var/lib/kubelet/pods/163b4b62-b2c8-4147-9e62-e0dfda89d7a2/volumes" Aug 02 10:04:42.393004 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:04:42.392966 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:06:12.888371 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:12.888286 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:06:20.494272 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:20.494237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:06:24.997539 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997506 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg"] Aug 02 10:06:24.998006 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997857 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" Aug 02 10:06:24.998006 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997868 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" Aug 02 10:06:24.998006 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997884 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" Aug 02 10:06:24.998006 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997890 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" Aug 02 10:06:24.998006 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997911 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" Aug 02 10:06:24.998006 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997917 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="295641e2-ccac-4b83-bcd6-eee4023dca5d" containerName="istio-proxy" Aug 02 10:06:24.998006 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997924 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" containerName="istio-proxy" Aug 02 10:06:24.998006 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997929 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" containerName="istio-proxy" Aug 02 10:06:24.998006 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997976 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" Aug 02 10:06:24.998006 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997983 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="163b4b62-b2c8-4147-9e62-e0dfda89d7a2" containerName="istio-proxy" Aug 02 10:06:24.998006 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:24.997990 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc219b53-c39d-4f19-bb4b-18e39e3a1bc9" containerName="istio-proxy" Aug 02 10:06:25.001399 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.001377 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.003800 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.003782 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 02 10:06:25.003919 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.003815 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-756eaa56-openshift-default-dockercfg-xdjp4\"" Aug 02 10:06:25.022282 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.022253 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg"] Aug 02 10:06:25.121887 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.121839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef99311-b183-4026-8c06-87d5b66a40b2-istiod-ca-cert\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.121887 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.121888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-credential-socket\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.122088 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.121945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktrn\" (UniqueName: \"kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-kube-api-access-7ktrn\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.122088 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.121967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-socket\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.122088 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.121984 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-envoy\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.122088 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.122058 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-data\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.122261 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.122103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-token\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.122261 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.122121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-certs\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.122261 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.122176 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-podinfo\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.222811 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.222782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-data\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.222976 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.222827 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-token\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.222976 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.222848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-certs\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.223060 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.222968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-podinfo\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.223060 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.223038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef99311-b183-4026-8c06-87d5b66a40b2-istiod-ca-cert\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.223153 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.223068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-credential-socket\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.223153 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.223139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktrn\" (UniqueName: \"kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-kube-api-access-7ktrn\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.223252 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.223194 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-data\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.223252 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.223238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-socket\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.223252 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.223244 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-certs\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.223429 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.223259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-envoy\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.223525 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.223503 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-credential-socket\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.223692 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.223668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-socket\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.223845 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.223823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef99311-b183-4026-8c06-87d5b66a40b2-istiod-ca-cert\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.225395 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.225377 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-envoy\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.225647 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.225625 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-podinfo\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.231136 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.231108 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-token\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.231319 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.231297 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktrn\" (UniqueName: \"kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-kube-api-access-7ktrn\") pod \"e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.312338 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.312247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:25.454817 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.454791 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg"] Aug 02 10:06:25.456578 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:06:25.456555 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef99311_b183_4026_8c06_87d5b66a40b2.slice/crio-bfc344a33b98365501d9bc9884d9ca66b418b66b25d1ebb7ee97e9d730e273fa WatchSource:0}: Error finding container bfc344a33b98365501d9bc9884d9ca66b418b66b25d1ebb7ee97e9d730e273fa: Status 404 returned error can't find the container with id bfc344a33b98365501d9bc9884d9ca66b418b66b25d1ebb7ee97e9d730e273fa Aug 02 10:06:25.458744 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.458708 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:06:25.458854 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.458788 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:06:25.458854 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.458831 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:06:25.505716 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:25.505682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" event={"ID":"9ef99311-b183-4026-8c06-87d5b66a40b2","Type":"ContainerStarted","Data":"bfc344a33b98365501d9bc9884d9ca66b418b66b25d1ebb7ee97e9d730e273fa"} Aug 02 10:06:26.512403 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:26.512368 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" event={"ID":"9ef99311-b183-4026-8c06-87d5b66a40b2","Type":"ContainerStarted","Data":"c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4"} Aug 02 10:06:26.534477 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:26.534410 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" podStartSLOduration=2.534390074 podStartE2EDuration="2.534390074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:06:26.531743285 +0000 UTC m=+2007.972633973" watchObservedRunningTime="2026-08-02 10:06:26.534390074 +0000 UTC m=+2007.975280532" Aug 02 10:06:27.313360 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:27.313328 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:27.318141 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:27.318117 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:27.518014 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:27.517982 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:27.518966 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:27.518946 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:32.781489 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:32.781428 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:06:42.536554 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:42.536521 2564 patch_prober.go:28] interesting pod/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:43014->10.134.0.39:15021: read: connection reset by peer" start-of-body= Aug 02 10:06:42.536971 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:42.536592 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:43014->10.134.0.39:15021: read: connection reset by peer" Aug 02 10:06:42.596806 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:42.596781 2564 generic.go:356] "Generic (PLEG): container finished" podID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerID="c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4" exitCode=0 Aug 02 10:06:42.596931 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:42.596863 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" event={"ID":"9ef99311-b183-4026-8c06-87d5b66a40b2","Type":"ContainerDied","Data":"c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4"} Aug 02 10:06:42.597296 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:42.597279 2564 scope.go:117] "RemoveContainer" containerID="c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4" Aug 02 10:06:42.598295 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:42.598266 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:06:42.598389 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:42.598356 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:06:42.598457 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:42.598408 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:06:43.604148 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:43.604109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" event={"ID":"9ef99311-b183-4026-8c06-87d5b66a40b2","Type":"ContainerStarted","Data":"5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053"} Aug 02 10:06:44.313000 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:44.312964 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:44.313210 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:44.313188 2564 patch_prober.go:28] interesting pod/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Aug 02 10:06:44.313260 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:44.313240 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Aug 02 10:06:45.313653 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:45.313616 2564 patch_prober.go:28] interesting pod/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Aug 02 10:06:45.314026 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:45.313672 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Aug 02 10:06:46.313254 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:46.313219 2564 patch_prober.go:28] interesting pod/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Aug 02 10:06:46.313430 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:46.313271 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Aug 02 10:06:47.312965 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:47.312925 2564 patch_prober.go:28] interesting pod/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Aug 02 10:06:47.313341 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:47.312989 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Aug 02 10:06:48.313127 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:48.313089 2564 patch_prober.go:28] interesting pod/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Aug 02 10:06:48.313512 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:48.313151 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Aug 02 10:06:49.317425 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:49.317394 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:49.317867 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:49.317680 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:06:49.318346 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:06:49.318329 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:07:13.190506 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:13.190457 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:07:17.733579 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:17.733546 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg"] Aug 02 10:07:17.734050 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:17.733752 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" containerID="cri-o://5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053" gracePeriod=30 Aug 02 10:07:19.320233 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:19.320187 2564 patch_prober.go:28] interesting pod/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 02 10:07:19.320632 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:19.320267 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 02 10:07:22.974881 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:22.974855 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:07:23.058840 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.058767 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-socket\") pod \"9ef99311-b183-4026-8c06-87d5b66a40b2\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " Aug 02 10:07:23.058840 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.058801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-credential-socket\") pod \"9ef99311-b183-4026-8c06-87d5b66a40b2\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " Aug 02 10:07:23.059056 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.058843 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ktrn\" (UniqueName: \"kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-kube-api-access-7ktrn\") pod \"9ef99311-b183-4026-8c06-87d5b66a40b2\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " Aug 02 10:07:23.059056 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.058892 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-envoy\") pod \"9ef99311-b183-4026-8c06-87d5b66a40b2\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " Aug 02 10:07:23.059056 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.058924 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-data\") pod \"9ef99311-b183-4026-8c06-87d5b66a40b2\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " Aug 02 10:07:23.059056 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.058947 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-certs\") pod \"9ef99311-b183-4026-8c06-87d5b66a40b2\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " Aug 02 10:07:23.059056 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.058977 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef99311-b183-4026-8c06-87d5b66a40b2-istiod-ca-cert\") pod \"9ef99311-b183-4026-8c06-87d5b66a40b2\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " Aug 02 10:07:23.059056 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.059007 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-token\") pod \"9ef99311-b183-4026-8c06-87d5b66a40b2\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " Aug 02 10:07:23.059056 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.059016 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ef99311-b183-4026-8c06-87d5b66a40b2" (UID: "9ef99311-b183-4026-8c06-87d5b66a40b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:07:23.059398 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.059066 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-podinfo\") pod \"9ef99311-b183-4026-8c06-87d5b66a40b2\" (UID: \"9ef99311-b183-4026-8c06-87d5b66a40b2\") " Aug 02 10:07:23.059398 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.059179 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ef99311-b183-4026-8c06-87d5b66a40b2" (UID: "9ef99311-b183-4026-8c06-87d5b66a40b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:07:23.059398 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.059196 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ef99311-b183-4026-8c06-87d5b66a40b2" (UID: "9ef99311-b183-4026-8c06-87d5b66a40b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:07:23.059398 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.059326 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ef99311-b183-4026-8c06-87d5b66a40b2" (UID: "9ef99311-b183-4026-8c06-87d5b66a40b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:07:23.059568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.059399 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:07:23.059568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.059421 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:07:23.059568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.059437 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:07:23.059672 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.059645 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef99311-b183-4026-8c06-87d5b66a40b2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ef99311-b183-4026-8c06-87d5b66a40b2" (UID: "9ef99311-b183-4026-8c06-87d5b66a40b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:07:23.061100 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.061071 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ef99311-b183-4026-8c06-87d5b66a40b2" (UID: "9ef99311-b183-4026-8c06-87d5b66a40b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:07:23.061228 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.061125 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-kube-api-access-7ktrn" (OuterVolumeSpecName: "kube-api-access-7ktrn") pod "9ef99311-b183-4026-8c06-87d5b66a40b2" (UID: "9ef99311-b183-4026-8c06-87d5b66a40b2"). InnerVolumeSpecName "kube-api-access-7ktrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:07:23.061428 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.061394 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ef99311-b183-4026-8c06-87d5b66a40b2" (UID: "9ef99311-b183-4026-8c06-87d5b66a40b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:07:23.061545 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.061454 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ef99311-b183-4026-8c06-87d5b66a40b2" (UID: "9ef99311-b183-4026-8c06-87d5b66a40b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:07:23.159955 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.159924 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktrn\" (UniqueName: \"kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-kube-api-access-7ktrn\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:07:23.159955 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.159950 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:07:23.159955 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.159961 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef99311-b183-4026-8c06-87d5b66a40b2-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:07:23.160164 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.159969 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef99311-b183-4026-8c06-87d5b66a40b2-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:07:23.160164 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.159977 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:07:23.160164 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.159985 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef99311-b183-4026-8c06-87d5b66a40b2-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:07:23.797483 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.797436 2564 generic.go:356] "Generic (PLEG): container finished" podID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerID="5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053" exitCode=0 Aug 02 10:07:23.797682 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.797497 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" event={"ID":"9ef99311-b183-4026-8c06-87d5b66a40b2","Type":"ContainerDied","Data":"5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053"} Aug 02 10:07:23.797682 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.797537 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" event={"ID":"9ef99311-b183-4026-8c06-87d5b66a40b2","Type":"ContainerDied","Data":"bfc344a33b98365501d9bc9884d9ca66b418b66b25d1ebb7ee97e9d730e273fa"} Aug 02 10:07:23.797682 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.797558 2564 scope.go:117] "RemoveContainer" containerID="5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053" Aug 02 10:07:23.797682 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.797499 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg" Aug 02 10:07:23.807598 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.807576 2564 scope.go:117] "RemoveContainer" containerID="c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4" Aug 02 10:07:23.815880 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.815862 2564 scope.go:117] "RemoveContainer" containerID="5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053" Aug 02 10:07:23.816146 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:07:23.816124 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053\": container with ID starting with 5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053 not found: ID does not exist" containerID="5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053" Aug 02 10:07:23.816190 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.816155 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053"} err="failed to get container status \"5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053\": rpc error: code = NotFound desc = could not find container \"5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053\": container with ID starting with 5c8c183189dfa27b0e5a524d055efe3335fcdc49e5e9ce5c3f1cc1b5401a7053 not found: ID does not exist" Aug 02 10:07:23.816190 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.816173 2564 scope.go:117] "RemoveContainer" containerID="c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4" Aug 02 10:07:23.816429 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:07:23.816408 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4\": container with ID starting with c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4 not found: ID does not exist" containerID="c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4" Aug 02 10:07:23.816503 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.816435 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4"} err="failed to get container status \"c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4\": rpc error: code = NotFound desc = could not find container \"c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4\": container with ID starting with c6adcdfdcd20ca4d516ff2757975329033fbf91c08d6bec42a07c26fe53a3fc4 not found: ID does not exist" Aug 02 10:07:23.820940 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.820915 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg"] Aug 02 10:07:23.825729 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:23.825706 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-756eaa56-openshift-default-745d697d5b-qfrcg"] Aug 02 10:07:24.978319 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:24.978285 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" path="/var/lib/kubelet/pods/9ef99311-b183-4026-8c06-87d5b66a40b2/volumes" Aug 02 10:07:49.093345 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:07:49.093304 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:08:59.153840 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:08:59.153779 2564 scope.go:117] "RemoveContainer" containerID="fb4ebf297ff46d59079987ebb8e3221cc755b8984dd9454a0537a2edce10fbf2" Aug 02 10:09:33.173247 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.173210 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2"] Aug 02 10:09:33.179291 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.179261 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" Aug 02 10:09:33.179291 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.179286 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" Aug 02 10:09:33.179291 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.179296 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" Aug 02 10:09:33.179518 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.179301 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" Aug 02 10:09:33.179518 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.179356 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" Aug 02 10:09:33.179518 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.179368 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef99311-b183-4026-8c06-87d5b66a40b2" containerName="istio-proxy" Aug 02 10:09:33.182674 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.182652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.185352 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.185332 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7e5ef7e7-openshift-default-dockercfg-qp27z\"" Aug 02 10:09:33.185718 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.185689 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 02 10:09:33.187253 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.187231 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2"] Aug 02 10:09:33.261989 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.261961 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvq8t\" (UniqueName: \"kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-kube-api-access-wvq8t\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.262100 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.261998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851e30bd-e939-4c41-a2ea-692751c444fa-istiod-ca-cert\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.262100 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.262023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-data\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.262100 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.262057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-credential-socket\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.262208 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.262097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-istio-token\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.262208 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.262163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-certs\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.262208 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.262193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-socket\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.262208 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.262207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-envoy\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.262340 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.262281 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851e30bd-e939-4c41-a2ea-692751c444fa-istio-podinfo\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.363387 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.363336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-certs\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.363561 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.363409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-socket\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.363561 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.363434 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-envoy\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.363561 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.363527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851e30bd-e939-4c41-a2ea-692751c444fa-istio-podinfo\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.363561 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.363554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvq8t\" (UniqueName: \"kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-kube-api-access-wvq8t\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.363768 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.363683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851e30bd-e939-4c41-a2ea-692751c444fa-istiod-ca-cert\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.363768 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.363715 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-data\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.363768 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.363748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-credential-socket\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.363962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.363783 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-istio-token\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.363962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.363801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-certs\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.363962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.363840 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-socket\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.364156 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.364126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-credential-socket\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.364378 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.364355 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-data\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.364526 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.364502 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851e30bd-e939-4c41-a2ea-692751c444fa-istiod-ca-cert\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.365798 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.365777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-envoy\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.366039 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.366018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851e30bd-e939-4c41-a2ea-692751c444fa-istio-podinfo\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.371398 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.371366 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-istio-token\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.371613 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.371593 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvq8t\" (UniqueName: \"kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-kube-api-access-wvq8t\") pod \"e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.495176 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.495151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:33.625360 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.625337 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2"] Aug 02 10:09:33.627618 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:09:33.627587 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851e30bd_e939_4c41_a2ea_692751c444fa.slice/crio-f8c4b503926a4b87071af55c3a99d0c2740a3b9aac6f5d9bdb55ee0a76718ef3 WatchSource:0}: Error finding container f8c4b503926a4b87071af55c3a99d0c2740a3b9aac6f5d9bdb55ee0a76718ef3: Status 404 returned error can't find the container with id f8c4b503926a4b87071af55c3a99d0c2740a3b9aac6f5d9bdb55ee0a76718ef3 Aug 02 10:09:33.630057 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.630014 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:09:33.630149 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.630107 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:09:33.630191 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:33.630154 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:09:34.429202 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:34.429171 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" event={"ID":"851e30bd-e939-4c41-a2ea-692751c444fa","Type":"ContainerStarted","Data":"2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a"} Aug 02 10:09:34.429626 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:34.429222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" event={"ID":"851e30bd-e939-4c41-a2ea-692751c444fa","Type":"ContainerStarted","Data":"f8c4b503926a4b87071af55c3a99d0c2740a3b9aac6f5d9bdb55ee0a76718ef3"} Aug 02 10:09:34.451373 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:34.451326 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" podStartSLOduration=1.4513117850000001 podStartE2EDuration="1.451311785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:09:34.448457269 +0000 UTC m=+2195.889347726" watchObservedRunningTime="2026-08-02 10:09:34.451311785 +0000 UTC m=+2195.892202240" Aug 02 10:09:34.496025 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:34.495994 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:34.500907 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:34.500884 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:35.085477 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:35.085420 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:09:35.435310 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:35.435228 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:35.436174 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:35.436153 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:43.782835 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:43.782792 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:09:50.436962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:50.436922 2564 patch_prober.go:28] interesting pod/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Aug 02 10:09:50.437381 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:50.437003 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Aug 02 10:09:50.513145 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:50.513107 2564 generic.go:356] "Generic (PLEG): container finished" podID="851e30bd-e939-4c41-a2ea-692751c444fa" containerID="2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a" exitCode=0 Aug 02 10:09:50.513323 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:50.513186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" event={"ID":"851e30bd-e939-4c41-a2ea-692751c444fa","Type":"ContainerDied","Data":"2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a"} Aug 02 10:09:50.513690 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:50.513672 2564 scope.go:117] "RemoveContainer" containerID="2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a" Aug 02 10:09:50.514744 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:50.514717 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:09:50.514807 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:50.514784 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:09:50.514849 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:50.514816 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:09:51.519663 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:51.519627 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" event={"ID":"851e30bd-e939-4c41-a2ea-692751c444fa","Type":"ContainerStarted","Data":"750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128"} Aug 02 10:09:52.496146 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:52.496101 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:52.496343 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:52.496316 2564 patch_prober.go:28] interesting pod/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Aug 02 10:09:52.496458 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:52.496388 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Aug 02 10:09:53.496093 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:53.496055 2564 patch_prober.go:28] interesting pod/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Aug 02 10:09:53.496499 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:53.496109 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Aug 02 10:09:54.495820 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:54.495785 2564 patch_prober.go:28] interesting pod/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Aug 02 10:09:54.495999 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:54.495837 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Aug 02 10:09:55.495586 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:55.495555 2564 patch_prober.go:28] interesting pod/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Aug 02 10:09:55.495958 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:55.495614 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Aug 02 10:09:56.495788 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:56.495751 2564 patch_prober.go:28] interesting pod/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Aug 02 10:09:56.496174 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:56.495810 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Aug 02 10:09:57.499713 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:57.499682 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:57.500194 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:57.500061 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:09:57.500674 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:09:57.500657 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:10:11.585559 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.585521 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4"] Aug 02 10:10:11.589685 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.589661 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.592694 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.592478 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7e5ef7e7\"/\"kube-root-ca.crt\"" Aug 02 10:10:11.593383 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.593361 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7e5ef7e7\"/\"e2e-lifecycle-model-7e5ef7e7-kserve-self-signed-certs\"" Aug 02 10:10:11.593574 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.593549 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7e5ef7e7\"/\"openshift-service-ca.crt\"" Aug 02 10:10:11.593689 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.593575 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7e5ef7e7\"/\"default-dockercfg-xxpj4\"" Aug 02 10:10:11.598621 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.598574 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4"] Aug 02 10:10:11.730304 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.730272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njcv5\" (UniqueName: \"kubernetes.io/projected/c579cfe2-f103-4ad5-845e-b098caa511dc-kube-api-access-njcv5\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.730501 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.730322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-dshm\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.730501 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.730340 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-model-cache\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.730501 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.730437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-home\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.730615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.730501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-tmp-dir\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.730615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.730559 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c579cfe2-f103-4ad5-845e-b098caa511dc-tls-certs\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.831622 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.831585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c579cfe2-f103-4ad5-845e-b098caa511dc-tls-certs\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.831781 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.831640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njcv5\" (UniqueName: \"kubernetes.io/projected/c579cfe2-f103-4ad5-845e-b098caa511dc-kube-api-access-njcv5\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.831781 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.831677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-dshm\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.831781 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.831691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-model-cache\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.831947 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.831861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-home\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.831947 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.831915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-tmp-dir\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.832168 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.832141 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-model-cache\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.832279 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.832177 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-home\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.832279 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.832270 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-tmp-dir\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.833956 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.833931 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-dshm\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.834128 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.834110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c579cfe2-f103-4ad5-845e-b098caa511dc-tls-certs\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.842525 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.842456 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njcv5\" (UniqueName: \"kubernetes.io/projected/c579cfe2-f103-4ad5-845e-b098caa511dc-kube-api-access-njcv5\") pod \"e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:11.902145 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:11.902116 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:12.029638 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:12.029610 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4"] Aug 02 10:10:12.032188 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:10:12.032158 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc579cfe2_f103_4ad5_845e_b098caa511dc.slice/crio-1a6c6b8e4e19fd542435c6ebc43751cb39a05e040b8de10ae32e97dc248f0255 WatchSource:0}: Error finding container 1a6c6b8e4e19fd542435c6ebc43751cb39a05e040b8de10ae32e97dc248f0255: Status 404 returned error can't find the container with id 1a6c6b8e4e19fd542435c6ebc43751cb39a05e040b8de10ae32e97dc248f0255 Aug 02 10:10:12.625778 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:12.625742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" event={"ID":"c579cfe2-f103-4ad5-845e-b098caa511dc","Type":"ContainerStarted","Data":"0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4"} Aug 02 10:10:12.625778 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:12.625788 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" event={"ID":"c579cfe2-f103-4ad5-845e-b098caa511dc","Type":"ContainerStarted","Data":"1a6c6b8e4e19fd542435c6ebc43751cb39a05e040b8de10ae32e97dc248f0255"} Aug 02 10:10:12.643056 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:12.643000 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" podStartSLOduration=1.642985216 podStartE2EDuration="1.642985216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:10:12.640867654 +0000 UTC m=+2234.081758111" watchObservedRunningTime="2026-08-02 10:10:12.642985216 +0000 UTC m=+2234.083875675" Aug 02 10:10:13.987993 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:13.987961 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:10:21.902685 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:21.902593 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:21.902685 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:21.902658 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:21.914960 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:21.914932 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:22.683168 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:22.683137 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:10:35.788065 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:10:35.788028 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:14:02.186249 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:02.186216 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:14:10.088865 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:10.088830 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4"] Aug 02 10:14:10.197079 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:10.197037 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:14:25.108118 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.108077 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" podUID="c579cfe2-f103-4ad5-845e-b098caa511dc" containerName="main" containerID="cri-o://0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4" gracePeriod=45 Aug 02 10:14:25.355510 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.355487 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:14:25.446558 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.446455 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-dshm\") pod \"c579cfe2-f103-4ad5-845e-b098caa511dc\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " Aug 02 10:14:25.446558 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.446515 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-model-cache\") pod \"c579cfe2-f103-4ad5-845e-b098caa511dc\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " Aug 02 10:14:25.446558 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.446539 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njcv5\" (UniqueName: \"kubernetes.io/projected/c579cfe2-f103-4ad5-845e-b098caa511dc-kube-api-access-njcv5\") pod \"c579cfe2-f103-4ad5-845e-b098caa511dc\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " Aug 02 10:14:25.446843 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.446602 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-home\") pod \"c579cfe2-f103-4ad5-845e-b098caa511dc\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " Aug 02 10:14:25.446843 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.446624 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-tmp-dir\") pod \"c579cfe2-f103-4ad5-845e-b098caa511dc\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " Aug 02 10:14:25.446843 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.446654 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c579cfe2-f103-4ad5-845e-b098caa511dc-tls-certs\") pod \"c579cfe2-f103-4ad5-845e-b098caa511dc\" (UID: \"c579cfe2-f103-4ad5-845e-b098caa511dc\") " Aug 02 10:14:25.446982 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.446842 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "c579cfe2-f103-4ad5-845e-b098caa511dc" (UID: "c579cfe2-f103-4ad5-845e-b098caa511dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:14:25.446982 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.446864 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-home" (OuterVolumeSpecName: "home") pod "c579cfe2-f103-4ad5-845e-b098caa511dc" (UID: "c579cfe2-f103-4ad5-845e-b098caa511dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:14:25.447068 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.447054 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:14:25.447104 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.447060 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c579cfe2-f103-4ad5-845e-b098caa511dc" (UID: "c579cfe2-f103-4ad5-845e-b098caa511dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:14:25.447104 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.447074 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:14:25.448739 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.448717 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-dshm" (OuterVolumeSpecName: "dshm") pod "c579cfe2-f103-4ad5-845e-b098caa511dc" (UID: "c579cfe2-f103-4ad5-845e-b098caa511dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:14:25.448739 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.448723 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c579cfe2-f103-4ad5-845e-b098caa511dc-kube-api-access-njcv5" (OuterVolumeSpecName: "kube-api-access-njcv5") pod "c579cfe2-f103-4ad5-845e-b098caa511dc" (UID: "c579cfe2-f103-4ad5-845e-b098caa511dc"). InnerVolumeSpecName "kube-api-access-njcv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:14:25.448739 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.448733 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c579cfe2-f103-4ad5-845e-b098caa511dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c579cfe2-f103-4ad5-845e-b098caa511dc" (UID: "c579cfe2-f103-4ad5-845e-b098caa511dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 10:14:25.548383 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.548358 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:14:25.548383 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.548380 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njcv5\" (UniqueName: \"kubernetes.io/projected/c579cfe2-f103-4ad5-845e-b098caa511dc-kube-api-access-njcv5\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:14:25.548568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.548407 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c579cfe2-f103-4ad5-845e-b098caa511dc-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:14:25.548568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.548419 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c579cfe2-f103-4ad5-845e-b098caa511dc-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:14:25.838770 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.838743 2564 generic.go:356] "Generic (PLEG): container finished" podID="c579cfe2-f103-4ad5-845e-b098caa511dc" containerID="0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4" exitCode=0 Aug 02 10:14:25.838919 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.838787 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" Aug 02 10:14:25.838919 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.838830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" event={"ID":"c579cfe2-f103-4ad5-845e-b098caa511dc","Type":"ContainerDied","Data":"0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4"} Aug 02 10:14:25.838919 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.838882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4" event={"ID":"c579cfe2-f103-4ad5-845e-b098caa511dc","Type":"ContainerDied","Data":"1a6c6b8e4e19fd542435c6ebc43751cb39a05e040b8de10ae32e97dc248f0255"} Aug 02 10:14:25.838919 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.838911 2564 scope.go:117] "RemoveContainer" containerID="0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4" Aug 02 10:14:25.848034 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.848026 2564 scope.go:117] "RemoveContainer" containerID="0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4" Aug 02 10:14:25.848277 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:14:25.848259 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4\": container with ID starting with 0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4 not found: ID does not exist" containerID="0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4" Aug 02 10:14:25.848329 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.848286 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4"} err="failed to get container status \"0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4\": rpc error: code = NotFound desc = could not find container \"0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4\": container with ID starting with 0ba1070f29d0fb0c6ed6619ad6d634004d73ad15964227235e0c8fef6ac252c4 not found: ID does not exist" Aug 02 10:14:25.864140 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.864117 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4"] Aug 02 10:14:25.876077 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:25.876057 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7e5ef7e7/e2e-lifecycle-model-7e5ef7e7-kserve-df99b55-qgcg4"] Aug 02 10:14:26.979120 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:14:26.979083 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c579cfe2-f103-4ad5-845e-b098caa511dc" path="/var/lib/kubelet/pods/c579cfe2-f103-4ad5-845e-b098caa511dc/volumes" Aug 02 10:15:09.620278 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:09.620237 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2"] Aug 02 10:15:09.620797 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:09.620506 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" containerID="cri-o://750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128" gracePeriod=30 Aug 02 10:15:11.758387 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:11.758352 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:15:12.502981 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:12.502935 2564 patch_prober.go:28] interesting pod/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 02 10:15:12.503154 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:12.502998 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 02 10:15:14.867506 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.867483 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:15:14.976795 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.976768 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-istio-token\") pod \"851e30bd-e939-4c41-a2ea-692751c444fa\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " Aug 02 10:15:14.976966 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.976815 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-data\") pod \"851e30bd-e939-4c41-a2ea-692751c444fa\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " Aug 02 10:15:14.976966 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.976847 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-credential-socket\") pod \"851e30bd-e939-4c41-a2ea-692751c444fa\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " Aug 02 10:15:14.976966 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.976889 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvq8t\" (UniqueName: \"kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-kube-api-access-wvq8t\") pod \"851e30bd-e939-4c41-a2ea-692751c444fa\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " Aug 02 10:15:14.976966 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.976905 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-socket\") pod \"851e30bd-e939-4c41-a2ea-692751c444fa\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " Aug 02 10:15:14.976966 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.976958 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851e30bd-e939-4c41-a2ea-692751c444fa-istiod-ca-cert\") pod \"851e30bd-e939-4c41-a2ea-692751c444fa\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " Aug 02 10:15:14.977241 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.976997 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851e30bd-e939-4c41-a2ea-692751c444fa-istio-podinfo\") pod \"851e30bd-e939-4c41-a2ea-692751c444fa\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " Aug 02 10:15:14.977241 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.977022 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-certs\") pod \"851e30bd-e939-4c41-a2ea-692751c444fa\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " Aug 02 10:15:14.977241 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.977049 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-envoy\") pod \"851e30bd-e939-4c41-a2ea-692751c444fa\" (UID: \"851e30bd-e939-4c41-a2ea-692751c444fa\") " Aug 02 10:15:14.977241 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.977224 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "851e30bd-e939-4c41-a2ea-692751c444fa" (UID: "851e30bd-e939-4c41-a2ea-692751c444fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:15:14.977535 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.977230 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "851e30bd-e939-4c41-a2ea-692751c444fa" (UID: "851e30bd-e939-4c41-a2ea-692751c444fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:15:14.977535 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.977375 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:15:14.977535 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.977394 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:15:14.977535 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.977454 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "851e30bd-e939-4c41-a2ea-692751c444fa" (UID: "851e30bd-e939-4c41-a2ea-692751c444fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:15:14.977753 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.977533 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "851e30bd-e939-4c41-a2ea-692751c444fa" (UID: "851e30bd-e939-4c41-a2ea-692751c444fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:15:14.977753 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.977678 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851e30bd-e939-4c41-a2ea-692751c444fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "851e30bd-e939-4c41-a2ea-692751c444fa" (UID: "851e30bd-e939-4c41-a2ea-692751c444fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:15:14.979327 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.979291 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/851e30bd-e939-4c41-a2ea-692751c444fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "851e30bd-e939-4c41-a2ea-692751c444fa" (UID: "851e30bd-e939-4c41-a2ea-692751c444fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:15:14.979457 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.979430 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "851e30bd-e939-4c41-a2ea-692751c444fa" (UID: "851e30bd-e939-4c41-a2ea-692751c444fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:15:14.979542 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.979430 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-kube-api-access-wvq8t" (OuterVolumeSpecName: "kube-api-access-wvq8t") pod "851e30bd-e939-4c41-a2ea-692751c444fa" (UID: "851e30bd-e939-4c41-a2ea-692751c444fa"). InnerVolumeSpecName "kube-api-access-wvq8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:15:14.979661 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:14.979644 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "851e30bd-e939-4c41-a2ea-692751c444fa" (UID: "851e30bd-e939-4c41-a2ea-692751c444fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:15:15.077844 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.077818 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvq8t\" (UniqueName: \"kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-kube-api-access-wvq8t\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:15:15.077844 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.077844 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:15:15.078046 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.077854 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851e30bd-e939-4c41-a2ea-692751c444fa-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:15:15.078046 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.077862 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851e30bd-e939-4c41-a2ea-692751c444fa-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:15:15.078046 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.077870 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:15:15.078046 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.077877 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851e30bd-e939-4c41-a2ea-692751c444fa-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:15:15.078046 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.077884 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851e30bd-e939-4c41-a2ea-692751c444fa-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:15:15.083242 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.083216 2564 generic.go:356] "Generic (PLEG): container finished" podID="851e30bd-e939-4c41-a2ea-692751c444fa" containerID="750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128" exitCode=0 Aug 02 10:15:15.083335 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.083272 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" Aug 02 10:15:15.083335 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.083291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" event={"ID":"851e30bd-e939-4c41-a2ea-692751c444fa","Type":"ContainerDied","Data":"750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128"} Aug 02 10:15:15.083411 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.083341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2" event={"ID":"851e30bd-e939-4c41-a2ea-692751c444fa","Type":"ContainerDied","Data":"f8c4b503926a4b87071af55c3a99d0c2740a3b9aac6f5d9bdb55ee0a76718ef3"} Aug 02 10:15:15.083411 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.083364 2564 scope.go:117] "RemoveContainer" containerID="750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128" Aug 02 10:15:15.092238 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.092222 2564 scope.go:117] "RemoveContainer" containerID="2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a" Aug 02 10:15:15.099931 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.099916 2564 scope.go:117] "RemoveContainer" containerID="750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128" Aug 02 10:15:15.100178 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:15:15.100161 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128\": container with ID starting with 750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128 not found: ID does not exist" containerID="750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128" Aug 02 10:15:15.100229 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.100186 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128"} err="failed to get container status \"750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128\": rpc error: code = NotFound desc = could not find container \"750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128\": container with ID starting with 750dc8a1f2d05d59fc2ef4337c30162f7bd9284db5cab976738a4a91f1de4128 not found: ID does not exist" Aug 02 10:15:15.100229 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.100208 2564 scope.go:117] "RemoveContainer" containerID="2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a" Aug 02 10:15:15.100442 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:15:15.100426 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a\": container with ID starting with 2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a not found: ID does not exist" containerID="2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a" Aug 02 10:15:15.100516 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.100447 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a"} err="failed to get container status \"2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a\": rpc error: code = NotFound desc = could not find container \"2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a\": container with ID starting with 2a1e77c3647fc17a9c6d74515f9d36ab111894ce35c04e4310aa747a26d12b5a not found: ID does not exist" Aug 02 10:15:15.114615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.114594 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2"] Aug 02 10:15:15.120402 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:15.120382 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7e5ef7e7-openshift-default-66dd786bf6-6gcl2"] Aug 02 10:15:16.978971 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:16.978927 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" path="/var/lib/kubelet/pods/851e30bd-e939-4c41-a2ea-692751c444fa/volumes" Aug 02 10:15:23.165416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:23.165380 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:15:30.944534 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:30.944497 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:15:39.211437 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:15:39.211402 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:16:23.358109 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.358026 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj"] Aug 02 10:16:23.358512 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.358423 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" Aug 02 10:16:23.358512 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.358435 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" Aug 02 10:16:23.358512 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.358444 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" Aug 02 10:16:23.358512 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.358449 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" Aug 02 10:16:23.358512 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.358456 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c579cfe2-f103-4ad5-845e-b098caa511dc" containerName="main" Aug 02 10:16:23.358512 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.358475 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c579cfe2-f103-4ad5-845e-b098caa511dc" containerName="main" Aug 02 10:16:23.358708 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.358550 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" Aug 02 10:16:23.358708 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.358565 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c579cfe2-f103-4ad5-845e-b098caa511dc" containerName="main" Aug 02 10:16:23.361836 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.361818 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.364408 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.364387 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 02 10:16:23.364703 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.364688 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e3a61d-openshift-default-dockercfg-6d2xt\"" Aug 02 10:16:23.377362 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.377339 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj"] Aug 02 10:16:23.492599 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.492566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-token\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.492784 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.492606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-podinfo\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.492784 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.492630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-envoy\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.492784 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.492646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istiod-ca-cert\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.492784 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.492713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmp7\" (UniqueName: \"kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-kube-api-access-jnmp7\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.492784 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.492762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-certs\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.492995 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.492800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-data\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.492995 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.492828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-socket\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.492995 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.492891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-credential-socket\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594131 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594106 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-certs\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594131 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-data\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594295 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-socket\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594295 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-credential-socket\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594295 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-token\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594295 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-podinfo\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594295 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-envoy\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594295 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istiod-ca-cert\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmp7\" (UniqueName: \"kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-kube-api-access-jnmp7\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594534 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-certs\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-socket\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594770 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594725 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-data\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.594815 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.594772 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-credential-socket\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.595140 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.595123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istiod-ca-cert\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.596653 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.596634 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-envoy\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.596880 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.596858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-podinfo\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.605067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.605047 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-token\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.608672 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.608610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmp7\" (UniqueName: \"kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-kube-api-access-jnmp7\") pod \"e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.671901 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.671872 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:23.807905 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.807879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj"] Aug 02 10:16:23.809833 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:16:23.809802 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bfaf5d_1539_4caf_a509_f4939db8dd70.slice/crio-507e6fb4909a613f9c58504b6404d2670f3dce5024bbf1c464482bf7fc265073 WatchSource:0}: Error finding container 507e6fb4909a613f9c58504b6404d2670f3dce5024bbf1c464482bf7fc265073: Status 404 returned error can't find the container with id 507e6fb4909a613f9c58504b6404d2670f3dce5024bbf1c464482bf7fc265073 Aug 02 10:16:23.811897 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.811858 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:16:23.811984 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.811925 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:16:23.811984 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:23.811952 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:16:24.424915 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:24.424882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" event={"ID":"f2bfaf5d-1539-4caf-a509-f4939db8dd70","Type":"ContainerStarted","Data":"1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace"} Aug 02 10:16:24.425270 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:24.424931 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" event={"ID":"f2bfaf5d-1539-4caf-a509-f4939db8dd70","Type":"ContainerStarted","Data":"507e6fb4909a613f9c58504b6404d2670f3dce5024bbf1c464482bf7fc265073"} Aug 02 10:16:24.451276 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:24.451229 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" podStartSLOduration=1.451213491 podStartE2EDuration="1.451213491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:16:24.449298293 +0000 UTC m=+2605.890188761" watchObservedRunningTime="2026-08-02 10:16:24.451213491 +0000 UTC m=+2605.892103946" Aug 02 10:16:24.672515 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:24.672482 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:24.677081 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:24.677013 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:25.429620 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:25.429584 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:25.430797 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:25.430776 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:30.640887 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:30.640853 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:16:40.431133 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:40.431038 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Aug 02 10:16:40.431133 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:40.431102 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Aug 02 10:16:40.501979 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:40.501946 2564 generic.go:356] "Generic (PLEG): container finished" podID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerID="1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace" exitCode=0 Aug 02 10:16:40.502116 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:40.502016 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" event={"ID":"f2bfaf5d-1539-4caf-a509-f4939db8dd70","Type":"ContainerDied","Data":"1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace"} Aug 02 10:16:40.502445 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:40.502429 2564 scope.go:117] "RemoveContainer" containerID="1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace" Aug 02 10:16:40.503473 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:40.503432 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:16:40.503569 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:40.503552 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:16:40.503610 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:40.503597 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:16:41.508241 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:41.508203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" event={"ID":"f2bfaf5d-1539-4caf-a509-f4939db8dd70","Type":"ContainerStarted","Data":"8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c"} Aug 02 10:16:41.672963 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:41.672920 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:41.673206 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:41.673181 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Aug 02 10:16:41.673269 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:41.673234 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Aug 02 10:16:42.672798 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:42.672763 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Aug 02 10:16:42.673186 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:42.672829 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Aug 02 10:16:43.673220 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:43.673180 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Aug 02 10:16:43.673629 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:43.673238 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Aug 02 10:16:44.673177 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:44.673139 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Aug 02 10:16:44.673357 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:44.673209 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Aug 02 10:16:45.672427 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:45.672387 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Aug 02 10:16:45.672623 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:45.672455 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Aug 02 10:16:46.676000 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:46.675969 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:46.676371 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:46.676226 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:46.676955 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:46.676936 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:16:52.142330 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.142296 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8"] Aug 02 10:16:52.142781 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.142766 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="851e30bd-e939-4c41-a2ea-692751c444fa" containerName="istio-proxy" Aug 02 10:16:52.144679 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.144662 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.147234 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.147215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6c8adc-openshift-default-dockercfg-6zmmc\"" Aug 02 10:16:52.170176 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.170151 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8"] Aug 02 10:16:52.269226 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.269194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04384b3-75c9-4d7e-9eac-76b09077d079-istiod-ca-cert\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.269226 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.269227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-credential-socket\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.269441 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.269262 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4vx\" (UniqueName: \"kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-kube-api-access-tk4vx\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.269441 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.269327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-token\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.269441 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.269361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-socket\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.269441 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.269381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-certs\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.269441 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.269405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-envoy\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.269441 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.269439 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-podinfo\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.269674 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.269532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-data\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370096 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04384b3-75c9-4d7e-9eac-76b09077d079-istiod-ca-cert\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370202 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370100 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-credential-socket\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370202 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4vx\" (UniqueName: \"kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-kube-api-access-tk4vx\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370202 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-token\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370355 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370204 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-socket\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370355 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-certs\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370457 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-envoy\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370457 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-podinfo\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370590 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-data\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370637 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-credential-socket\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370684 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370638 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-certs\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370886 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-data\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.370886 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370874 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04384b3-75c9-4d7e-9eac-76b09077d079-istiod-ca-cert\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.371033 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.370921 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-socket\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.372673 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.372652 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-envoy\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.372817 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.372798 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-podinfo\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.378361 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.378338 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-token\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.378869 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.378848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4vx\" (UniqueName: \"kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-kube-api-access-tk4vx\") pod \"e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.454865 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.454840 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:52.588757 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.588731 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8"] Aug 02 10:16:52.590856 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:16:52.590815 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04384b3_75c9_4d7e_9eac_76b09077d079.slice/crio-bb588bca54264ba37fe68b49c556e2d98f022586cd44ef41f993cb208bb3b733 WatchSource:0}: Error finding container bb588bca54264ba37fe68b49c556e2d98f022586cd44ef41f993cb208bb3b733: Status 404 returned error can't find the container with id bb588bca54264ba37fe68b49c556e2d98f022586cd44ef41f993cb208bb3b733 Aug 02 10:16:52.592927 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.592895 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:16:52.592998 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.592957 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:16:52.592998 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:52.592985 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:16:53.567440 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:53.567406 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" event={"ID":"b04384b3-75c9-4d7e-9eac-76b09077d079","Type":"ContainerStarted","Data":"2ac2fd482438953eca1d5a3a3bb95c1c2a7fd175d8924d76efcad5ab18bc8c89"} Aug 02 10:16:53.567798 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:53.567456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" event={"ID":"b04384b3-75c9-4d7e-9eac-76b09077d079","Type":"ContainerStarted","Data":"bb588bca54264ba37fe68b49c556e2d98f022586cd44ef41f993cb208bb3b733"} Aug 02 10:16:53.591253 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:53.591206 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" podStartSLOduration=1.591190794 podStartE2EDuration="1.591190794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:16:53.589688342 +0000 UTC m=+2635.030578803" watchObservedRunningTime="2026-08-02 10:16:53.591190794 +0000 UTC m=+2635.032081250" Aug 02 10:16:54.455056 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:54.455023 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:54.459948 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:54.459921 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:54.572726 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:54.572695 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:54.573647 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:54.573627 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:16:59.508152 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:16:59.508120 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:17:09.573444 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:09.573403 2564 patch_prober.go:28] interesting pod/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Aug 02 10:17:09.573873 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:09.573495 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Aug 02 10:17:09.655408 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:09.655378 2564 generic.go:356] "Generic (PLEG): container finished" podID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerID="2ac2fd482438953eca1d5a3a3bb95c1c2a7fd175d8924d76efcad5ab18bc8c89" exitCode=0 Aug 02 10:17:09.655568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:09.655451 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" event={"ID":"b04384b3-75c9-4d7e-9eac-76b09077d079","Type":"ContainerDied","Data":"2ac2fd482438953eca1d5a3a3bb95c1c2a7fd175d8924d76efcad5ab18bc8c89"} Aug 02 10:17:09.655846 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:09.655830 2564 scope.go:117] "RemoveContainer" containerID="2ac2fd482438953eca1d5a3a3bb95c1c2a7fd175d8924d76efcad5ab18bc8c89" Aug 02 10:17:09.656943 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:09.656913 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:17:09.657050 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:09.656989 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:17:09.657050 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:09.657020 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:17:10.664742 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:10.664706 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" event={"ID":"b04384b3-75c9-4d7e-9eac-76b09077d079","Type":"ContainerStarted","Data":"640cf4895fa0bc872181f52a215876cd6f646e2e970eee7baf8d380306933624"} Aug 02 10:17:11.455828 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:11.455783 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:17:11.456315 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:11.456285 2564 patch_prober.go:28] interesting pod/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Aug 02 10:17:11.456423 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:11.456353 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Aug 02 10:17:12.456121 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:12.456079 2564 patch_prober.go:28] interesting pod/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Aug 02 10:17:12.456724 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:12.456690 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Aug 02 10:17:13.456667 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:13.456170 2564 patch_prober.go:28] interesting pod/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Aug 02 10:17:13.456667 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:13.456236 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Aug 02 10:17:14.455666 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:14.455629 2564 patch_prober.go:28] interesting pod/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Aug 02 10:17:14.455960 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:14.455928 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Aug 02 10:17:15.456552 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:15.456041 2564 patch_prober.go:28] interesting pod/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Aug 02 10:17:15.456552 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:15.456122 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Aug 02 10:17:16.460059 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:16.460027 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:17:16.460564 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:16.460410 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:17:16.461192 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:16.461169 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:17:29.093329 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.093298 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp"] Aug 02 10:17:29.095768 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.095744 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.098216 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.098198 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e3a61d\"/\"kube-root-ca.crt\"" Aug 02 10:17:29.098911 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.098884 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e3a61d\"/\"openshift-service-ca.crt\"" Aug 02 10:17:29.099022 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.099000 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e3a61d\"/\"default-dockercfg-2tflj\"" Aug 02 10:17:29.099088 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.099035 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e3a61d\"/\"test-model-e3a61d-kserve-self-signed-certs\"" Aug 02 10:17:29.108249 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.108225 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp"] Aug 02 10:17:29.212634 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.212603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-home\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.212792 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.212644 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-model-cache\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.212792 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.212669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tmp-dir\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.212792 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.212729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k26j\" (UniqueName: \"kubernetes.io/projected/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-kube-api-access-9k26j\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.212792 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.212782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-dshm\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.212953 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.212874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tls-certs\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.314008 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.313979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tls-certs\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.314176 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.314014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-home\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.314176 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.314044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-model-cache\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.314285 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.314174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tmp-dir\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.314285 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.314234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k26j\" (UniqueName: \"kubernetes.io/projected/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-kube-api-access-9k26j\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.314392 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.314289 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-dshm\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.314490 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.314453 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-model-cache\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.314579 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.314555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-home\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.314642 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.314577 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tmp-dir\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.316526 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.316493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-dshm\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.316643 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.316604 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tls-certs\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.323826 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.323805 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k26j\" (UniqueName: \"kubernetes.io/projected/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-kube-api-access-9k26j\") pod \"test-model-e3a61d-kserve-6d5bb8b57c-tq2hp\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.404817 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.404760 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:29.536832 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.536805 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp"] Aug 02 10:17:29.538711 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:17:29.538683 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43a1089b_0e2a_4a36_ac74_5ddf8b378dfb.slice/crio-002fb7ed84b373ddd1800a0ac352a4f3a1e930550d5537c62fddb5d74bc4247a WatchSource:0}: Error finding container 002fb7ed84b373ddd1800a0ac352a4f3a1e930550d5537c62fddb5d74bc4247a: Status 404 returned error can't find the container with id 002fb7ed84b373ddd1800a0ac352a4f3a1e930550d5537c62fddb5d74bc4247a Aug 02 10:17:29.761600 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.761567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" event={"ID":"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb","Type":"ContainerStarted","Data":"a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6"} Aug 02 10:17:29.761762 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.761608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" event={"ID":"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb","Type":"ContainerStarted","Data":"002fb7ed84b373ddd1800a0ac352a4f3a1e930550d5537c62fddb5d74bc4247a"} Aug 02 10:17:29.779802 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:29.779753 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" podStartSLOduration=0.779736501 podStartE2EDuration="779.736501ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:17:29.778293228 +0000 UTC m=+2671.219183684" watchObservedRunningTime="2026-08-02 10:17:29.779736501 +0000 UTC m=+2671.220627020" Aug 02 10:17:31.802270 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:31.802238 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:17:38.705346 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:38.705287 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:17:39.404884 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:39.404845 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:39.405051 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:39.404898 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:39.417262 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:39.417236 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:17:39.819825 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:17:39.819798 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:18:07.000038 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.000000 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc"] Aug 02 10:18:07.004001 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.003977 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.006440 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.006408 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6c8adc\"/\"kube-root-ca.crt\"" Aug 02 10:18:07.006560 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.006508 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6c8adc\"/\"test-model-6c8adc-kserve-self-signed-certs\"" Aug 02 10:18:07.006560 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.006513 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6c8adc\"/\"openshift-service-ca.crt\"" Aug 02 10:18:07.006672 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.006578 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6c8adc\"/\"default-dockercfg-mxshh\"" Aug 02 10:18:07.014580 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.014556 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc"] Aug 02 10:18:07.042295 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.042274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-model-cache\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.042429 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.042340 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-dshm\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.042429 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.042395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jz2m\" (UniqueName: \"kubernetes.io/projected/98948d45-1827-424f-84f2-b2f772c52484-kube-api-access-5jz2m\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.042569 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.042428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-tmp-dir\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.042569 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.042473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-home\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.042569 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.042527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98948d45-1827-424f-84f2-b2f772c52484-tls-certs\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.143819 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.143788 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-model-cache\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.143946 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.143860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-dshm\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.143946 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.143917 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jz2m\" (UniqueName: \"kubernetes.io/projected/98948d45-1827-424f-84f2-b2f772c52484-kube-api-access-5jz2m\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.144024 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.143947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-tmp-dir\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.144024 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.143978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-home\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.144024 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.144013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98948d45-1827-424f-84f2-b2f772c52484-tls-certs\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.144244 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.144218 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-model-cache\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.144328 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.144308 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-tmp-dir\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.144375 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.144331 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-home\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.146170 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.146153 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-dshm\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.146266 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.146250 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98948d45-1827-424f-84f2-b2f772c52484-tls-certs\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.152650 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.152627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jz2m\" (UniqueName: \"kubernetes.io/projected/98948d45-1827-424f-84f2-b2f772c52484-kube-api-access-5jz2m\") pod \"test-model-6c8adc-kserve-847c586d7f-lhmsc\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.313918 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.313838 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:07.447180 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.447153 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc"] Aug 02 10:18:07.449443 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:18:07.449412 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98948d45_1827_424f_84f2_b2f772c52484.slice/crio-f97bf55f94e762850c036fbcae47eafb22c425b4d35ca12c7b0dc39f7596a1a5 WatchSource:0}: Error finding container f97bf55f94e762850c036fbcae47eafb22c425b4d35ca12c7b0dc39f7596a1a5: Status 404 returned error can't find the container with id f97bf55f94e762850c036fbcae47eafb22c425b4d35ca12c7b0dc39f7596a1a5 Aug 02 10:18:07.950964 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.950930 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" event={"ID":"98948d45-1827-424f-84f2-b2f772c52484","Type":"ContainerStarted","Data":"1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254"} Aug 02 10:18:07.951126 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.950972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" event={"ID":"98948d45-1827-424f-84f2-b2f772c52484","Type":"ContainerStarted","Data":"f97bf55f94e762850c036fbcae47eafb22c425b4d35ca12c7b0dc39f7596a1a5"} Aug 02 10:18:07.968318 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:07.968273 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" podStartSLOduration=1.9682558239999999 podStartE2EDuration="1.968255824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:18:07.967623005 +0000 UTC m=+2709.408513462" watchObservedRunningTime="2026-08-02 10:18:07.968255824 +0000 UTC m=+2709.409146280" Aug 02 10:18:11.006456 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:11.002959 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:18:17.314397 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:17.314359 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:17.314820 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:17.314409 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:17.326916 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:17.326884 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:18.015628 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:18.015598 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:18:55.735449 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:55.735415 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp"] Aug 02 10:18:56.826548 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:56.826508 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc"] Aug 02 10:18:59.486628 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:18:59.486597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:19:06.891793 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:06.891756 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:19:10.755235 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:10.755190 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" podUID="43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" containerName="main" containerID="cri-o://a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6" gracePeriod=45 Aug 02 10:19:11.003893 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.003869 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:19:11.130487 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.130387 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-model-cache\") pod \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " Aug 02 10:19:11.130487 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.130437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tmp-dir\") pod \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " Aug 02 10:19:11.130706 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.130495 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k26j\" (UniqueName: \"kubernetes.io/projected/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-kube-api-access-9k26j\") pod \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " Aug 02 10:19:11.130706 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.130549 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tls-certs\") pod \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " Aug 02 10:19:11.130706 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.130617 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-dshm\") pod \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " Aug 02 10:19:11.130706 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.130663 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-home\") pod \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\" (UID: \"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb\") " Aug 02 10:19:11.130706 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.130670 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-model-cache" (OuterVolumeSpecName: "model-cache") pod "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" (UID: "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:11.130962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.130702 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" (UID: "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:11.131017 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.130979 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-home" (OuterVolumeSpecName: "home") pod "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" (UID: "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:11.131017 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.131002 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:11.131099 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.131022 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:11.132783 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.132756 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" (UID: "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 10:19:11.132900 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.132820 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-kube-api-access-9k26j" (OuterVolumeSpecName: "kube-api-access-9k26j") pod "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" (UID: "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb"). InnerVolumeSpecName "kube-api-access-9k26j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:19:11.132900 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.132839 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-dshm" (OuterVolumeSpecName: "dshm") pod "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" (UID: "43a1089b-0e2a-4a36-ac74-5ddf8b378dfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:11.231749 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.231725 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k26j\" (UniqueName: \"kubernetes.io/projected/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-kube-api-access-9k26j\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:11.231749 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.231747 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:11.231904 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.231757 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:11.231904 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.231765 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:11.272272 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.272243 2564 generic.go:356] "Generic (PLEG): container finished" podID="43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" containerID="a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6" exitCode=0 Aug 02 10:19:11.272392 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.272297 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" Aug 02 10:19:11.272392 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.272336 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" event={"ID":"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb","Type":"ContainerDied","Data":"a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6"} Aug 02 10:19:11.272392 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.272387 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp" event={"ID":"43a1089b-0e2a-4a36-ac74-5ddf8b378dfb","Type":"ContainerDied","Data":"002fb7ed84b373ddd1800a0ac352a4f3a1e930550d5537c62fddb5d74bc4247a"} Aug 02 10:19:11.272537 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.272410 2564 scope.go:117] "RemoveContainer" containerID="a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6" Aug 02 10:19:11.281591 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.281576 2564 scope.go:117] "RemoveContainer" containerID="a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6" Aug 02 10:19:11.281832 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:19:11.281807 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6\": container with ID starting with a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6 not found: ID does not exist" containerID="a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6" Aug 02 10:19:11.281926 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.281835 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6"} err="failed to get container status \"a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6\": rpc error: code = NotFound desc = could not find container \"a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6\": container with ID starting with a08fb22081a00141e487c2dea2aac5c2d0704200c08bf53269ebcc5f4dac4db6 not found: ID does not exist" Aug 02 10:19:11.294779 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.294755 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp"] Aug 02 10:19:11.299114 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.299094 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e3a61d/test-model-e3a61d-kserve-6d5bb8b57c-tq2hp"] Aug 02 10:19:11.845767 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:11.845731 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" podUID="98948d45-1827-424f-84f2-b2f772c52484" containerName="main" containerID="cri-o://1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254" gracePeriod=45 Aug 02 10:19:12.084021 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.084000 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:19:12.242594 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.242564 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-home\") pod \"98948d45-1827-424f-84f2-b2f772c52484\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " Aug 02 10:19:12.242743 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.242641 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-tmp-dir\") pod \"98948d45-1827-424f-84f2-b2f772c52484\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " Aug 02 10:19:12.242743 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.242711 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98948d45-1827-424f-84f2-b2f772c52484-tls-certs\") pod \"98948d45-1827-424f-84f2-b2f772c52484\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " Aug 02 10:19:12.242743 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.242733 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-dshm\") pod \"98948d45-1827-424f-84f2-b2f772c52484\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " Aug 02 10:19:12.242933 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.242762 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-model-cache\") pod \"98948d45-1827-424f-84f2-b2f772c52484\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " Aug 02 10:19:12.242933 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.242766 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-home" (OuterVolumeSpecName: "home") pod "98948d45-1827-424f-84f2-b2f772c52484" (UID: "98948d45-1827-424f-84f2-b2f772c52484"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:12.242933 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.242810 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jz2m\" (UniqueName: \"kubernetes.io/projected/98948d45-1827-424f-84f2-b2f772c52484-kube-api-access-5jz2m\") pod \"98948d45-1827-424f-84f2-b2f772c52484\" (UID: \"98948d45-1827-424f-84f2-b2f772c52484\") " Aug 02 10:19:12.242933 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.242923 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98948d45-1827-424f-84f2-b2f772c52484" (UID: "98948d45-1827-424f-84f2-b2f772c52484"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:12.243118 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.243055 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-model-cache" (OuterVolumeSpecName: "model-cache") pod "98948d45-1827-424f-84f2-b2f772c52484" (UID: "98948d45-1827-424f-84f2-b2f772c52484"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:12.243193 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.243172 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:12.243193 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.243194 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:12.243348 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.243209 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:12.244881 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.244861 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-dshm" (OuterVolumeSpecName: "dshm") pod "98948d45-1827-424f-84f2-b2f772c52484" (UID: "98948d45-1827-424f-84f2-b2f772c52484"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:12.244962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.244896 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98948d45-1827-424f-84f2-b2f772c52484-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98948d45-1827-424f-84f2-b2f772c52484" (UID: "98948d45-1827-424f-84f2-b2f772c52484"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 10:19:12.245002 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.244965 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98948d45-1827-424f-84f2-b2f772c52484-kube-api-access-5jz2m" (OuterVolumeSpecName: "kube-api-access-5jz2m") pod "98948d45-1827-424f-84f2-b2f772c52484" (UID: "98948d45-1827-424f-84f2-b2f772c52484"). InnerVolumeSpecName "kube-api-access-5jz2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:19:12.278943 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.278915 2564 generic.go:356] "Generic (PLEG): container finished" podID="98948d45-1827-424f-84f2-b2f772c52484" containerID="1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254" exitCode=0 Aug 02 10:19:12.279058 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.278960 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" Aug 02 10:19:12.279058 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.279004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" event={"ID":"98948d45-1827-424f-84f2-b2f772c52484","Type":"ContainerDied","Data":"1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254"} Aug 02 10:19:12.279058 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.279046 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc" event={"ID":"98948d45-1827-424f-84f2-b2f772c52484","Type":"ContainerDied","Data":"f97bf55f94e762850c036fbcae47eafb22c425b4d35ca12c7b0dc39f7596a1a5"} Aug 02 10:19:12.279204 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.279063 2564 scope.go:117] "RemoveContainer" containerID="1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254" Aug 02 10:19:12.287620 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.287603 2564 scope.go:117] "RemoveContainer" containerID="1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254" Aug 02 10:19:12.287892 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:19:12.287874 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254\": container with ID starting with 1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254 not found: ID does not exist" containerID="1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254" Aug 02 10:19:12.287956 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.287904 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254"} err="failed to get container status \"1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254\": rpc error: code = NotFound desc = could not find container \"1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254\": container with ID starting with 1c7746529d5a72c176b2d5017e4c0c33d758e6c248632d2dca113dfc78cd4254 not found: ID does not exist" Aug 02 10:19:12.318686 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.318664 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc"] Aug 02 10:19:12.322202 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.322176 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6c8adc/test-model-6c8adc-kserve-847c586d7f-lhmsc"] Aug 02 10:19:12.343995 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.343970 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jz2m\" (UniqueName: \"kubernetes.io/projected/98948d45-1827-424f-84f2-b2f772c52484-kube-api-access-5jz2m\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:12.343995 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.343992 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98948d45-1827-424f-84f2-b2f772c52484-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:12.344128 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.344002 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98948d45-1827-424f-84f2-b2f772c52484-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:12.979080 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.979036 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" path="/var/lib/kubelet/pods/43a1089b-0e2a-4a36-ac74-5ddf8b378dfb/volumes" Aug 02 10:19:12.979659 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:12.979637 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98948d45-1827-424f-84f2-b2f772c52484" path="/var/lib/kubelet/pods/98948d45-1827-424f-84f2-b2f772c52484/volumes" Aug 02 10:19:13.519413 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:13.519380 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj"] Aug 02 10:19:13.519630 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:13.519607 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" containerID="cri-o://8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c" gracePeriod=30 Aug 02 10:19:16.678832 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:16.678799 2564 patch_prober.go:28] interesting pod/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 02 10:19:16.679186 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:16.678855 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 02 10:19:18.769244 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.769218 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:19:18.908137 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908069 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istiod-ca-cert\") pod \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " Aug 02 10:19:18.908137 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908109 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-token\") pod \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " Aug 02 10:19:18.908308 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908141 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-data\") pod \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " Aug 02 10:19:18.908308 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908258 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-credential-socket\") pod \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " Aug 02 10:19:18.908402 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908321 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnmp7\" (UniqueName: \"kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-kube-api-access-jnmp7\") pod \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " Aug 02 10:19:18.908402 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908360 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-certs\") pod \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " Aug 02 10:19:18.908402 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908375 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2bfaf5d-1539-4caf-a509-f4939db8dd70" (UID: "f2bfaf5d-1539-4caf-a509-f4939db8dd70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:18.908570 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908452 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-socket\") pod \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " Aug 02 10:19:18.908570 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908508 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-podinfo\") pod \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " Aug 02 10:19:18.908570 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908519 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2bfaf5d-1539-4caf-a509-f4939db8dd70" (UID: "f2bfaf5d-1539-4caf-a509-f4939db8dd70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:19:18.908570 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908541 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-envoy\") pod \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\" (UID: \"f2bfaf5d-1539-4caf-a509-f4939db8dd70\") " Aug 02 10:19:18.908780 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908537 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2bfaf5d-1539-4caf-a509-f4939db8dd70" (UID: "f2bfaf5d-1539-4caf-a509-f4939db8dd70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:18.908836 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908732 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2bfaf5d-1539-4caf-a509-f4939db8dd70" (UID: "f2bfaf5d-1539-4caf-a509-f4939db8dd70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:18.908836 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908775 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2bfaf5d-1539-4caf-a509-f4939db8dd70" (UID: "f2bfaf5d-1539-4caf-a509-f4939db8dd70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:18.908921 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908885 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:18.908921 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908903 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:18.908921 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908916 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:18.909043 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908929 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:18.909043 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.908941 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:18.910365 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.910341 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2bfaf5d-1539-4caf-a509-f4939db8dd70" (UID: "f2bfaf5d-1539-4caf-a509-f4939db8dd70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:19:18.910636 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.910617 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2bfaf5d-1539-4caf-a509-f4939db8dd70" (UID: "f2bfaf5d-1539-4caf-a509-f4939db8dd70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:19:18.910875 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.910855 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2bfaf5d-1539-4caf-a509-f4939db8dd70" (UID: "f2bfaf5d-1539-4caf-a509-f4939db8dd70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:18.910962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:18.910900 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-kube-api-access-jnmp7" (OuterVolumeSpecName: "kube-api-access-jnmp7") pod "f2bfaf5d-1539-4caf-a509-f4939db8dd70" (UID: "f2bfaf5d-1539-4caf-a509-f4939db8dd70"). InnerVolumeSpecName "kube-api-access-jnmp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:19:19.009660 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.009634 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:19.009660 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.009657 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:19.009843 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.009668 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:19.009843 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.009676 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmp7\" (UniqueName: \"kubernetes.io/projected/f2bfaf5d-1539-4caf-a509-f4939db8dd70-kube-api-access-jnmp7\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:19.316651 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.316616 2564 generic.go:356] "Generic (PLEG): container finished" podID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerID="8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c" exitCode=0 Aug 02 10:19:19.316816 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.316673 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" Aug 02 10:19:19.316816 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.316704 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" event={"ID":"f2bfaf5d-1539-4caf-a509-f4939db8dd70","Type":"ContainerDied","Data":"8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c"} Aug 02 10:19:19.316816 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.316762 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj" event={"ID":"f2bfaf5d-1539-4caf-a509-f4939db8dd70","Type":"ContainerDied","Data":"507e6fb4909a613f9c58504b6404d2670f3dce5024bbf1c464482bf7fc265073"} Aug 02 10:19:19.316816 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.316784 2564 scope.go:117] "RemoveContainer" containerID="8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c" Aug 02 10:19:19.325622 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.325607 2564 scope.go:117] "RemoveContainer" containerID="1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace" Aug 02 10:19:19.333399 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.333380 2564 scope.go:117] "RemoveContainer" containerID="8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c" Aug 02 10:19:19.333685 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:19:19.333665 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c\": container with ID starting with 8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c not found: ID does not exist" containerID="8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c" Aug 02 10:19:19.333761 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.333699 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c"} err="failed to get container status \"8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c\": rpc error: code = NotFound desc = could not find container \"8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c\": container with ID starting with 8737084923ad0a0732fd10664702af0ec487f93bf0c12d63a8bbd2835172153c not found: ID does not exist" Aug 02 10:19:19.333761 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.333723 2564 scope.go:117] "RemoveContainer" containerID="1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace" Aug 02 10:19:19.333966 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:19:19.333949 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace\": container with ID starting with 1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace not found: ID does not exist" containerID="1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace" Aug 02 10:19:19.334009 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.333972 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace"} err="failed to get container status \"1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace\": rpc error: code = NotFound desc = could not find container \"1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace\": container with ID starting with 1d7e3579b0496b2dca3d8dcf79f8d9946a5fe52c81fb1e1504574c2a02c41ace not found: ID does not exist" Aug 02 10:19:19.338051 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.338029 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj"] Aug 02 10:19:19.341554 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:19.341531 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e3a61d-openshift-default-7d9bb58cf8-lkknj"] Aug 02 10:19:20.979043 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:20.979006 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" path="/var/lib/kubelet/pods/f2bfaf5d-1539-4caf-a509-f4939db8dd70/volumes" Aug 02 10:19:26.490759 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:26.490720 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:19:31.191037 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:31.190998 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8"] Aug 02 10:19:31.191576 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:31.191252 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" containerID="cri-o://640cf4895fa0bc872181f52a215876cd6f646e2e970eee7baf8d380306933624" gracePeriod=30 Aug 02 10:19:31.462817 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:31.462782 2564 patch_prober.go:28] interesting pod/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 02 10:19:31.462975 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:31.462854 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 02 10:19:32.145103 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145068 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc"] Aug 02 10:19:32.145481 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145449 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98948d45-1827-424f-84f2-b2f772c52484" containerName="main" Aug 02 10:19:32.145481 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145480 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="98948d45-1827-424f-84f2-b2f772c52484" containerName="main" Aug 02 10:19:32.145616 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145507 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" Aug 02 10:19:32.145616 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145513 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" Aug 02 10:19:32.145616 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145520 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" containerName="main" Aug 02 10:19:32.145616 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145526 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" containerName="main" Aug 02 10:19:32.145616 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145536 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" Aug 02 10:19:32.145616 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145541 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" Aug 02 10:19:32.145616 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145612 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" Aug 02 10:19:32.145827 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145622 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="43a1089b-0e2a-4a36-ac74-5ddf8b378dfb" containerName="main" Aug 02 10:19:32.145827 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145629 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="98948d45-1827-424f-84f2-b2f772c52484" containerName="main" Aug 02 10:19:32.145827 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.145636 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2bfaf5d-1539-4caf-a509-f4939db8dd70" containerName="istio-proxy" Aug 02 10:19:32.150608 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.150591 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.155389 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.155371 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-931e8a-openshift-default-dockercfg-wr67d\"" Aug 02 10:19:32.179051 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.179025 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc"] Aug 02 10:19:32.212034 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.212006 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istiod-ca-cert\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.212354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.212036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-token\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.212354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.212066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-socket\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.212354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.212162 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-envoy\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.212354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.212193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59466\" (UniqueName: \"kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-kube-api-access-59466\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.212354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.212223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-data\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.212354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.212262 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-podinfo\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.212354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.212313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-credential-socket\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.212354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.212352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-certs\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.312830 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.312789 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-socket\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313021 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.312870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-envoy\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313021 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.312902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59466\" (UniqueName: \"kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-kube-api-access-59466\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313021 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.312931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-data\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313021 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.312956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-podinfo\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313021 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.313009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-credential-socket\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313293 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.313044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-certs\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313293 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.313105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istiod-ca-cert\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313293 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.313134 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-token\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313293 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.313241 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-socket\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313510 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.313361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-credential-socket\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313590 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.313518 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-data\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313767 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.313738 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-certs\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.313890 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.313832 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istiod-ca-cert\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.315209 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.315188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-envoy\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.315519 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.315500 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-podinfo\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.321109 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.321089 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-token\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.321486 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.321448 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59466\" (UniqueName: \"kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-kube-api-access-59466\") pod \"e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.460596 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.460573 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:32.603233 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.603197 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc"] Aug 02 10:19:32.607794 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:19:32.607771 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0423f2_6d41_4cf5_95c3_3927cad974e2.slice/crio-d73c31ec53577a75a2e7749739a4a40475f623c561b2a735432a373151decf39 WatchSource:0}: Error finding container d73c31ec53577a75a2e7749739a4a40475f623c561b2a735432a373151decf39: Status 404 returned error can't find the container with id d73c31ec53577a75a2e7749739a4a40475f623c561b2a735432a373151decf39 Aug 02 10:19:32.610292 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.610259 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:19:32.610350 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.610334 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:19:32.610391 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:32.610362 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:19:33.386937 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:33.386897 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" event={"ID":"7b0423f2-6d41-4cf5-95c3-3927cad974e2","Type":"ContainerStarted","Data":"b75dbb3389904063df8afcac8edd59aca49015383dbfe94ef28d4cf9d515ffaf"} Aug 02 10:19:33.387297 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:33.386964 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" event={"ID":"7b0423f2-6d41-4cf5-95c3-3927cad974e2","Type":"ContainerStarted","Data":"d73c31ec53577a75a2e7749739a4a40475f623c561b2a735432a373151decf39"} Aug 02 10:19:33.416345 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:33.416299 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" podStartSLOduration=1.416285716 podStartE2EDuration="1.416285716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:19:33.414716798 +0000 UTC m=+2794.855607255" watchObservedRunningTime="2026-08-02 10:19:33.416285716 +0000 UTC m=+2794.857176171" Aug 02 10:19:33.460770 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:33.460743 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:33.465535 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:33.465513 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:34.391830 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:34.391795 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:34.392646 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:34.392630 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:36.405820 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.405787 2564 generic.go:356] "Generic (PLEG): container finished" podID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerID="640cf4895fa0bc872181f52a215876cd6f646e2e970eee7baf8d380306933624" exitCode=0 Aug 02 10:19:36.406264 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.405858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" event={"ID":"b04384b3-75c9-4d7e-9eac-76b09077d079","Type":"ContainerDied","Data":"640cf4895fa0bc872181f52a215876cd6f646e2e970eee7baf8d380306933624"} Aug 02 10:19:36.406264 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.405914 2564 scope.go:117] "RemoveContainer" containerID="2ac2fd482438953eca1d5a3a3bb95c1c2a7fd175d8924d76efcad5ab18bc8c89" Aug 02 10:19:36.440159 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.440135 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:19:36.552478 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.552437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-envoy\") pod \"b04384b3-75c9-4d7e-9eac-76b09077d079\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " Aug 02 10:19:36.552648 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.552497 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-data\") pod \"b04384b3-75c9-4d7e-9eac-76b09077d079\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " Aug 02 10:19:36.552648 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.552537 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-credential-socket\") pod \"b04384b3-75c9-4d7e-9eac-76b09077d079\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " Aug 02 10:19:36.552648 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.552562 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-token\") pod \"b04384b3-75c9-4d7e-9eac-76b09077d079\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " Aug 02 10:19:36.552648 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.552606 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-podinfo\") pod \"b04384b3-75c9-4d7e-9eac-76b09077d079\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " Aug 02 10:19:36.552882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.552724 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-socket\") pod \"b04384b3-75c9-4d7e-9eac-76b09077d079\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " Aug 02 10:19:36.552882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.552751 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04384b3-75c9-4d7e-9eac-76b09077d079-istiod-ca-cert\") pod \"b04384b3-75c9-4d7e-9eac-76b09077d079\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " Aug 02 10:19:36.552882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.552782 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-certs\") pod \"b04384b3-75c9-4d7e-9eac-76b09077d079\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " Aug 02 10:19:36.552882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.552815 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk4vx\" (UniqueName: \"kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-kube-api-access-tk4vx\") pod \"b04384b3-75c9-4d7e-9eac-76b09077d079\" (UID: \"b04384b3-75c9-4d7e-9eac-76b09077d079\") " Aug 02 10:19:36.552882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.552815 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b04384b3-75c9-4d7e-9eac-76b09077d079" (UID: "b04384b3-75c9-4d7e-9eac-76b09077d079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:36.552882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.552830 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-data" (OuterVolumeSpecName: "istio-data") pod "b04384b3-75c9-4d7e-9eac-76b09077d079" (UID: "b04384b3-75c9-4d7e-9eac-76b09077d079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:36.553184 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.553060 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b04384b3-75c9-4d7e-9eac-76b09077d079" (UID: "b04384b3-75c9-4d7e-9eac-76b09077d079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:36.553184 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.553079 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:36.553184 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.553096 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:36.553184 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.553153 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04384b3-75c9-4d7e-9eac-76b09077d079-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b04384b3-75c9-4d7e-9eac-76b09077d079" (UID: "b04384b3-75c9-4d7e-9eac-76b09077d079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:19:36.553377 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.553270 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b04384b3-75c9-4d7e-9eac-76b09077d079" (UID: "b04384b3-75c9-4d7e-9eac-76b09077d079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:36.554786 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.554760 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b04384b3-75c9-4d7e-9eac-76b09077d079" (UID: "b04384b3-75c9-4d7e-9eac-76b09077d079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:19:36.554786 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.554777 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-token" (OuterVolumeSpecName: "istio-token") pod "b04384b3-75c9-4d7e-9eac-76b09077d079" (UID: "b04384b3-75c9-4d7e-9eac-76b09077d079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:19:36.554786 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.554760 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b04384b3-75c9-4d7e-9eac-76b09077d079" (UID: "b04384b3-75c9-4d7e-9eac-76b09077d079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:19:36.554982 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.554895 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-kube-api-access-tk4vx" (OuterVolumeSpecName: "kube-api-access-tk4vx") pod "b04384b3-75c9-4d7e-9eac-76b09077d079" (UID: "b04384b3-75c9-4d7e-9eac-76b09077d079"). InnerVolumeSpecName "kube-api-access-tk4vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:19:36.654365 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.654338 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk4vx\" (UniqueName: \"kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-kube-api-access-tk4vx\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:36.654365 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.654363 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:36.654365 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.654372 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:36.654605 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.654381 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b04384b3-75c9-4d7e-9eac-76b09077d079-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:36.654605 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.654388 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:36.654605 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.654398 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b04384b3-75c9-4d7e-9eac-76b09077d079-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:36.654605 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:36.654408 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04384b3-75c9-4d7e-9eac-76b09077d079-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:19:37.414836 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:37.414756 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" Aug 02 10:19:37.414836 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:37.414776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8" event={"ID":"b04384b3-75c9-4d7e-9eac-76b09077d079","Type":"ContainerDied","Data":"bb588bca54264ba37fe68b49c556e2d98f022586cd44ef41f993cb208bb3b733"} Aug 02 10:19:37.414836 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:37.414828 2564 scope.go:117] "RemoveContainer" containerID="640cf4895fa0bc872181f52a215876cd6f646e2e970eee7baf8d380306933624" Aug 02 10:19:37.431650 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:37.431628 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8"] Aug 02 10:19:37.435741 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:37.435720 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6c8adc-openshift-default-7c857fb8fd-8fdr8"] Aug 02 10:19:38.979000 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:38.978967 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" path="/var/lib/kubelet/pods/b04384b3-75c9-4d7e-9eac-76b09077d079/volumes" Aug 02 10:19:39.791275 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:39.791240 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:19:49.421755 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:49.421139 2564 patch_prober.go:28] interesting pod/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": read tcp 10.134.0.2:55714->10.134.0.46:15021: read: connection reset by peer" start-of-body= Aug 02 10:19:49.421755 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:49.421213 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": read tcp 10.134.0.2:55714->10.134.0.46:15021: read: connection reset by peer" Aug 02 10:19:49.491606 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:49.491576 2564 generic.go:356] "Generic (PLEG): container finished" podID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerID="b75dbb3389904063df8afcac8edd59aca49015383dbfe94ef28d4cf9d515ffaf" exitCode=0 Aug 02 10:19:49.491753 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:49.491661 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" event={"ID":"7b0423f2-6d41-4cf5-95c3-3927cad974e2","Type":"ContainerDied","Data":"b75dbb3389904063df8afcac8edd59aca49015383dbfe94ef28d4cf9d515ffaf"} Aug 02 10:19:49.492085 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:49.492070 2564 scope.go:117] "RemoveContainer" containerID="b75dbb3389904063df8afcac8edd59aca49015383dbfe94ef28d4cf9d515ffaf" Aug 02 10:19:49.493075 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:49.493050 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:19:49.493135 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:49.493116 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:19:49.493176 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:49.493149 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:19:50.498334 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:50.498304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" event={"ID":"7b0423f2-6d41-4cf5-95c3-3927cad974e2","Type":"ContainerStarted","Data":"a55e521c1b2d87541604c1ec5dc7cd9150b6ade36fca0bf4ba5831ff452defe5"} Aug 02 10:19:51.461423 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:51.461384 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:51.461637 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:51.461452 2564 patch_prober.go:28] interesting pod/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Aug 02 10:19:51.461637 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:51.461523 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Aug 02 10:19:52.461705 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:52.461666 2564 patch_prober.go:28] interesting pod/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Aug 02 10:19:52.462101 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:52.461723 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Aug 02 10:19:53.461602 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:53.461561 2564 patch_prober.go:28] interesting pod/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Aug 02 10:19:53.461785 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:53.461616 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Aug 02 10:19:54.461537 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:54.461495 2564 patch_prober.go:28] interesting pod/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Aug 02 10:19:54.461710 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:54.461559 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Aug 02 10:19:55.465081 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:55.465038 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:55.465598 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:55.465449 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:19:55.466124 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:19:55.466102 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:20:01.163224 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.163192 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf"] Aug 02 10:20:01.163615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.163601 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" Aug 02 10:20:01.163615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.163613 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" Aug 02 10:20:01.163696 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.163629 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" Aug 02 10:20:01.163696 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.163634 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" Aug 02 10:20:01.163766 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.163701 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" Aug 02 10:20:01.168441 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.168422 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.174377 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.174353 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2ea423-openshift-default-dockercfg-2rnzh\"" Aug 02 10:20:01.183960 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.183939 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf"] Aug 02 10:20:01.212348 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.212318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-istio-token\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.212503 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.212395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f932cd-405e-44fb-87bd-8691fc407440-istiod-ca-cert\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.212503 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.212431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-socket\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.212503 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.212496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-data\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.212638 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.212543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p56n6\" (UniqueName: \"kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-kube-api-access-p56n6\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.212638 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.212584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f932cd-405e-44fb-87bd-8691fc407440-istio-podinfo\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.212743 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.212723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-certs\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.212780 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.212747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-envoy\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.212780 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.212770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-credential-socket\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.313611 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.313568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f932cd-405e-44fb-87bd-8691fc407440-istio-podinfo\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.313823 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.313654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-certs\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.313823 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.313672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-envoy\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.313823 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.313691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-credential-socket\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.313823 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.313730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-istio-token\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.313823 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.313770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f932cd-405e-44fb-87bd-8691fc407440-istiod-ca-cert\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.313823 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.313801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-socket\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.314114 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.313852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-data\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.314114 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.313881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p56n6\" (UniqueName: \"kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-kube-api-access-p56n6\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.314235 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.314110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-certs\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.314235 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.314148 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-credential-socket\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.314389 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.314360 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-socket\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.314492 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.314417 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-data\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.314842 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.314815 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f932cd-405e-44fb-87bd-8691fc407440-istiod-ca-cert\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.316169 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.316148 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-envoy\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.316309 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.316291 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f932cd-405e-44fb-87bd-8691fc407440-istio-podinfo\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.322873 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.322853 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-istio-token\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.322979 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.322928 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p56n6\" (UniqueName: \"kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-kube-api-access-p56n6\") pod \"e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.478656 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.478627 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:01.616212 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.616183 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf"] Aug 02 10:20:01.616915 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:20:01.616888 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f932cd_405e_44fb_87bd_8691fc407440.slice/crio-3fb7537e479e26266b32691b2f0143d03e8fec6b18b0fff61dbda1adcfaeb2b1 WatchSource:0}: Error finding container 3fb7537e479e26266b32691b2f0143d03e8fec6b18b0fff61dbda1adcfaeb2b1: Status 404 returned error can't find the container with id 3fb7537e479e26266b32691b2f0143d03e8fec6b18b0fff61dbda1adcfaeb2b1 Aug 02 10:20:01.619108 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.619066 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:20:01.619176 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.619157 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:20:01.619227 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:01.619212 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:20:02.558815 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:02.558778 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" event={"ID":"e2f932cd-405e-44fb-87bd-8691fc407440","Type":"ContainerStarted","Data":"402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3"} Aug 02 10:20:02.559403 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:02.558828 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" event={"ID":"e2f932cd-405e-44fb-87bd-8691fc407440","Type":"ContainerStarted","Data":"3fb7537e479e26266b32691b2f0143d03e8fec6b18b0fff61dbda1adcfaeb2b1"} Aug 02 10:20:02.587444 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:02.587379 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" podStartSLOduration=1.587364382 podStartE2EDuration="1.587364382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:20:02.587065686 +0000 UTC m=+2824.027956153" watchObservedRunningTime="2026-08-02 10:20:02.587364382 +0000 UTC m=+2824.028254838" Aug 02 10:20:03.479013 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:03.478972 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:03.483848 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:03.483822 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:03.563640 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:03.563607 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:03.564577 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:03.564556 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:08.205909 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:08.205874 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:20:17.632228 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:17.632142 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2f932cd-405e-44fb-87bd-8691fc407440" containerID="402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3" exitCode=0 Aug 02 10:20:17.632763 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:17.632218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" event={"ID":"e2f932cd-405e-44fb-87bd-8691fc407440","Type":"ContainerDied","Data":"402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3"} Aug 02 10:20:17.632839 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:17.632776 2564 scope.go:117] "RemoveContainer" containerID="402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3" Aug 02 10:20:17.633862 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:17.633834 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:20:17.633969 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:17.633913 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:20:17.633969 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:17.633960 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:20:18.564553 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:18.564503 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:18.639308 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:18.639266 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" event={"ID":"e2f932cd-405e-44fb-87bd-8691fc407440","Type":"ContainerStarted","Data":"61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11"} Aug 02 10:20:19.478846 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:19.478807 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:19.479917 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:19.479889 2564 patch_prober.go:28] interesting pod/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Aug 02 10:20:19.480049 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:19.479951 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Aug 02 10:20:20.479594 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:20.479559 2564 patch_prober.go:28] interesting pod/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Aug 02 10:20:20.480007 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:20.479624 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Aug 02 10:20:21.479899 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:21.479859 2564 patch_prober.go:28] interesting pod/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Aug 02 10:20:21.480280 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:21.479919 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Aug 02 10:20:22.479785 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:22.479743 2564 patch_prober.go:28] interesting pod/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Aug 02 10:20:22.479962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:22.479804 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Aug 02 10:20:23.479484 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:23.479426 2564 patch_prober.go:28] interesting pod/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Aug 02 10:20:23.479667 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:23.479503 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Aug 02 10:20:24.483325 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:24.483291 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:24.483727 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:24.483563 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:24.484282 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:24.484262 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:20:50.787140 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:50.787104 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:20:54.656993 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:54.656959 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc"] Aug 02 10:20:54.657431 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:54.657204 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" containerID="cri-o://a55e521c1b2d87541604c1ec5dc7cd9150b6ade36fca0bf4ba5831ff452defe5" gracePeriod=30 Aug 02 10:20:55.467831 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:55.467791 2564 patch_prober.go:28] interesting pod/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 02 10:20:55.467992 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:55.467849 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 02 10:20:59.849754 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.849721 2564 generic.go:356] "Generic (PLEG): container finished" podID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerID="a55e521c1b2d87541604c1ec5dc7cd9150b6ade36fca0bf4ba5831ff452defe5" exitCode=0 Aug 02 10:20:59.850098 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.849768 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" event={"ID":"7b0423f2-6d41-4cf5-95c3-3927cad974e2","Type":"ContainerDied","Data":"a55e521c1b2d87541604c1ec5dc7cd9150b6ade36fca0bf4ba5831ff452defe5"} Aug 02 10:20:59.850098 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.849891 2564 scope.go:117] "RemoveContainer" containerID="b75dbb3389904063df8afcac8edd59aca49015383dbfe94ef28d4cf9d515ffaf" Aug 02 10:20:59.903559 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.903537 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:20:59.957405 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.957383 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-certs\") pod \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " Aug 02 10:20:59.957559 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.957417 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-podinfo\") pod \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " Aug 02 10:20:59.957559 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.957437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59466\" (UniqueName: \"kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-kube-api-access-59466\") pod \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " Aug 02 10:20:59.957559 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.957507 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-envoy\") pod \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " Aug 02 10:20:59.957559 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.957555 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-data\") pod \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " Aug 02 10:20:59.957779 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.957593 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-credential-socket\") pod \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " Aug 02 10:20:59.957779 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.957621 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-socket\") pod \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " Aug 02 10:20:59.957779 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.957658 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b0423f2-6d41-4cf5-95c3-3927cad974e2" (UID: "7b0423f2-6d41-4cf5-95c3-3927cad974e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:20:59.957779 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.957673 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istiod-ca-cert\") pod \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " Aug 02 10:20:59.957779 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.957760 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-token\") pod \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\" (UID: \"7b0423f2-6d41-4cf5-95c3-3927cad974e2\") " Aug 02 10:20:59.958024 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.957948 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b0423f2-6d41-4cf5-95c3-3927cad974e2" (UID: "7b0423f2-6d41-4cf5-95c3-3927cad974e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:20:59.958081 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.958057 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b0423f2-6d41-4cf5-95c3-3927cad974e2" (UID: "7b0423f2-6d41-4cf5-95c3-3927cad974e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:20:59.958136 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.958083 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b0423f2-6d41-4cf5-95c3-3927cad974e2" (UID: "7b0423f2-6d41-4cf5-95c3-3927cad974e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:20:59.958136 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.958101 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b0423f2-6d41-4cf5-95c3-3927cad974e2" (UID: "7b0423f2-6d41-4cf5-95c3-3927cad974e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:20:59.958247 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.958226 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:20:59.958316 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.958276 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:20:59.958316 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.958292 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:20:59.958316 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.958304 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:20:59.958495 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.958316 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:20:59.959646 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.959622 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b0423f2-6d41-4cf5-95c3-3927cad974e2" (UID: "7b0423f2-6d41-4cf5-95c3-3927cad974e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:20:59.959883 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.959860 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b0423f2-6d41-4cf5-95c3-3927cad974e2" (UID: "7b0423f2-6d41-4cf5-95c3-3927cad974e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:20:59.959974 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.959911 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-kube-api-access-59466" (OuterVolumeSpecName: "kube-api-access-59466") pod "7b0423f2-6d41-4cf5-95c3-3927cad974e2" (UID: "7b0423f2-6d41-4cf5-95c3-3927cad974e2"). InnerVolumeSpecName "kube-api-access-59466". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:20:59.960070 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:20:59.960052 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b0423f2-6d41-4cf5-95c3-3927cad974e2" (UID: "7b0423f2-6d41-4cf5-95c3-3927cad974e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:21:00.059657 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:00.059620 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:00.059657 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:00.059650 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:00.059657 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:00.059663 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59466\" (UniqueName: \"kubernetes.io/projected/7b0423f2-6d41-4cf5-95c3-3927cad974e2-kube-api-access-59466\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:00.059890 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:00.059676 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0423f2-6d41-4cf5-95c3-3927cad974e2-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:00.856268 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:00.856235 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" event={"ID":"7b0423f2-6d41-4cf5-95c3-3927cad974e2","Type":"ContainerDied","Data":"d73c31ec53577a75a2e7749739a4a40475f623c561b2a735432a373151decf39"} Aug 02 10:21:00.856638 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:00.856290 2564 scope.go:117] "RemoveContainer" containerID="a55e521c1b2d87541604c1ec5dc7cd9150b6ade36fca0bf4ba5831ff452defe5" Aug 02 10:21:00.856638 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:00.856248 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc" Aug 02 10:21:00.879972 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:00.879947 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc"] Aug 02 10:21:00.885954 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:00.885932 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-931e8a-openshift-default-bfd6df5c8-bf8sc"] Aug 02 10:21:00.978100 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:00.978071 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" path="/var/lib/kubelet/pods/7b0423f2-6d41-4cf5-95c3-3927cad974e2/volumes" Aug 02 10:21:12.596147 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:12.596106 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:21:17.334163 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:17.334128 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf"] Aug 02 10:21:17.334545 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:17.334386 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" containerID="cri-o://61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11" gracePeriod=30 Aug 02 10:21:18.254583 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.254543 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm"] Aug 02 10:21:18.255022 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.255006 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" Aug 02 10:21:18.255078 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.255024 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" Aug 02 10:21:18.255078 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.255041 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" Aug 02 10:21:18.255078 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.255047 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" Aug 02 10:21:18.255169 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.255116 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" Aug 02 10:21:18.255169 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.255124 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b04384b3-75c9-4d7e-9eac-76b09077d079" containerName="istio-proxy" Aug 02 10:21:18.255169 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.255130 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0423f2-6d41-4cf5-95c3-3927cad974e2" containerName="istio-proxy" Aug 02 10:21:18.257235 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.257216 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.259587 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.259570 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e9225c-openshift-default-dockercfg-qbcqd\"" Aug 02 10:21:18.275226 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.275201 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm"] Aug 02 10:21:18.430213 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.430185 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-socket\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.430599 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.430223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istiod-ca-cert\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.430599 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.430254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-token\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.430599 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.430342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-certs\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.430599 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.430380 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwcss\" (UniqueName: \"kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-kube-api-access-dwcss\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.430599 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.430438 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-data\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.430599 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.430459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-credential-socket\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.430599 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.430507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-podinfo\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.430599 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.430525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-envoy\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.531444 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.531367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwcss\" (UniqueName: \"kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-kube-api-access-dwcss\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.531444 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.531419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-data\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.531444 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.531443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-credential-socket\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.531704 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.531491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-podinfo\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.531704 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.531520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-envoy\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.531704 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.531558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-socket\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.531704 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.531594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istiod-ca-cert\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.531704 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.531643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-token\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.531963 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.531708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-certs\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.531963 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.531858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-data\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.531963 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.531890 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-credential-socket\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.532130 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.532044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-socket\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.532130 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.532050 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-certs\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.532408 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.532390 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istiod-ca-cert\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.533935 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.533913 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-envoy\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.534135 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.534118 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-podinfo\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.541344 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.541319 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-token\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.541447 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.541392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwcss\" (UniqueName: \"kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-kube-api-access-dwcss\") pod \"e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.568408 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.568386 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:18.785049 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.785024 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm"] Aug 02 10:21:18.787246 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:21:18.787217 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b20eaa_3045_46a2_93c3_1ebefd00ea1d.slice/crio-9814b4204ff3e499f81acebf6ae7437f43ba809632fe9d659d25ecfbe8db50e7 WatchSource:0}: Error finding container 9814b4204ff3e499f81acebf6ae7437f43ba809632fe9d659d25ecfbe8db50e7: Status 404 returned error can't find the container with id 9814b4204ff3e499f81acebf6ae7437f43ba809632fe9d659d25ecfbe8db50e7 Aug 02 10:21:18.789371 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.789327 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:21:18.789437 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.789410 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:21:18.789487 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.789437 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:21:18.944618 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.944578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" event={"ID":"10b20eaa-3045-46a2-93c3-1ebefd00ea1d","Type":"ContainerStarted","Data":"3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7"} Aug 02 10:21:18.944924 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.944643 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" event={"ID":"10b20eaa-3045-46a2-93c3-1ebefd00ea1d","Type":"ContainerStarted","Data":"9814b4204ff3e499f81acebf6ae7437f43ba809632fe9d659d25ecfbe8db50e7"} Aug 02 10:21:18.970362 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:18.970313 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" podStartSLOduration=0.970297596 podStartE2EDuration="970.297596ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:21:18.968044982 +0000 UTC m=+2900.408935439" watchObservedRunningTime="2026-08-02 10:21:18.970297596 +0000 UTC m=+2900.411188052" Aug 02 10:21:19.569277 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:19.569243 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:19.574218 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:19.574193 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:19.949002 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:19.948928 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:19.949760 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:19.949743 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:21:22.590618 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.590593 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:21:22.667377 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.667306 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-data\") pod \"e2f932cd-405e-44fb-87bd-8691fc407440\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " Aug 02 10:21:22.667377 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.667342 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-certs\") pod \"e2f932cd-405e-44fb-87bd-8691fc407440\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " Aug 02 10:21:22.667377 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.667379 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-socket\") pod \"e2f932cd-405e-44fb-87bd-8691fc407440\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " Aug 02 10:21:22.667636 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.667396 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p56n6\" (UniqueName: \"kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-kube-api-access-p56n6\") pod \"e2f932cd-405e-44fb-87bd-8691fc407440\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " Aug 02 10:21:22.667636 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.667445 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f932cd-405e-44fb-87bd-8691fc407440-istiod-ca-cert\") pod \"e2f932cd-405e-44fb-87bd-8691fc407440\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " Aug 02 10:21:22.667636 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.667558 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2f932cd-405e-44fb-87bd-8691fc407440" (UID: "e2f932cd-405e-44fb-87bd-8691fc407440"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:21:22.667926 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.667866 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2f932cd-405e-44fb-87bd-8691fc407440" (UID: "e2f932cd-405e-44fb-87bd-8691fc407440"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:21:22.668072 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.667483 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f932cd-405e-44fb-87bd-8691fc407440-istio-podinfo\") pod \"e2f932cd-405e-44fb-87bd-8691fc407440\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " Aug 02 10:21:22.668072 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.668044 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-credential-socket\") pod \"e2f932cd-405e-44fb-87bd-8691fc407440\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " Aug 02 10:21:22.668189 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.668072 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2f932cd-405e-44fb-87bd-8691fc407440" (UID: "e2f932cd-405e-44fb-87bd-8691fc407440"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:21:22.668189 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.668089 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-envoy\") pod \"e2f932cd-405e-44fb-87bd-8691fc407440\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " Aug 02 10:21:22.668189 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.668146 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-istio-token\") pod \"e2f932cd-405e-44fb-87bd-8691fc407440\" (UID: \"e2f932cd-405e-44fb-87bd-8691fc407440\") " Aug 02 10:21:22.668189 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.668161 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f932cd-405e-44fb-87bd-8691fc407440-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2f932cd-405e-44fb-87bd-8691fc407440" (UID: "e2f932cd-405e-44fb-87bd-8691fc407440"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:21:22.668389 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.668273 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2f932cd-405e-44fb-87bd-8691fc407440" (UID: "e2f932cd-405e-44fb-87bd-8691fc407440"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:21:22.676746 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.670559 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f932cd-405e-44fb-87bd-8691fc407440-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:22.676746 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.670587 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:22.676746 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.670602 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:22.676746 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.670625 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:22.676746 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.670638 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:22.677297 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.677243 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2f932cd-405e-44fb-87bd-8691fc407440-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2f932cd-405e-44fb-87bd-8691fc407440" (UID: "e2f932cd-405e-44fb-87bd-8691fc407440"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:21:22.677373 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.677331 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2f932cd-405e-44fb-87bd-8691fc407440" (UID: "e2f932cd-405e-44fb-87bd-8691fc407440"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:21:22.677437 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.677416 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-kube-api-access-p56n6" (OuterVolumeSpecName: "kube-api-access-p56n6") pod "e2f932cd-405e-44fb-87bd-8691fc407440" (UID: "e2f932cd-405e-44fb-87bd-8691fc407440"). InnerVolumeSpecName "kube-api-access-p56n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:21:22.678149 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.678121 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2f932cd-405e-44fb-87bd-8691fc407440" (UID: "e2f932cd-405e-44fb-87bd-8691fc407440"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:21:22.771564 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.771532 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:22.771564 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.771559 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p56n6\" (UniqueName: \"kubernetes.io/projected/e2f932cd-405e-44fb-87bd-8691fc407440-kube-api-access-p56n6\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:22.771564 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.771570 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f932cd-405e-44fb-87bd-8691fc407440-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:22.771776 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.771579 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f932cd-405e-44fb-87bd-8691fc407440-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:21:22.965591 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.965559 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2f932cd-405e-44fb-87bd-8691fc407440" containerID="61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11" exitCode=0 Aug 02 10:21:22.965767 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.965627 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" Aug 02 10:21:22.965767 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.965650 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" event={"ID":"e2f932cd-405e-44fb-87bd-8691fc407440","Type":"ContainerDied","Data":"61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11"} Aug 02 10:21:22.965767 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.965740 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf" event={"ID":"e2f932cd-405e-44fb-87bd-8691fc407440","Type":"ContainerDied","Data":"3fb7537e479e26266b32691b2f0143d03e8fec6b18b0fff61dbda1adcfaeb2b1"} Aug 02 10:21:22.965942 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.965771 2564 scope.go:117] "RemoveContainer" containerID="61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11" Aug 02 10:21:22.975053 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.975038 2564 scope.go:117] "RemoveContainer" containerID="402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3" Aug 02 10:21:22.984313 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.984296 2564 scope.go:117] "RemoveContainer" containerID="61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11" Aug 02 10:21:22.984593 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:21:22.984572 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11\": container with ID starting with 61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11 not found: ID does not exist" containerID="61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11" Aug 02 10:21:22.984684 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.984603 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11"} err="failed to get container status \"61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11\": rpc error: code = NotFound desc = could not find container \"61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11\": container with ID starting with 61df544d5336e138075465fc2cb867a33fe8ffe11ecff59d6ca84f26984a8d11 not found: ID does not exist" Aug 02 10:21:22.984684 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.984626 2564 scope.go:117] "RemoveContainer" containerID="402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3" Aug 02 10:21:22.984893 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:21:22.984876 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3\": container with ID starting with 402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3 not found: ID does not exist" containerID="402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3" Aug 02 10:21:22.984929 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.984899 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3"} err="failed to get container status \"402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3\": rpc error: code = NotFound desc = could not find container \"402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3\": container with ID starting with 402b9acd81b84dcc9aedcda968e667c18f6b5b88d5462118c2bab82be86464b3 not found: ID does not exist" Aug 02 10:21:22.990761 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.990737 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf"] Aug 02 10:21:22.992768 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:22.992749 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2ea423-openshift-default-84b7f6b684-xpskf"] Aug 02 10:21:24.696751 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:24.696718 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:21:24.979201 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:24.979166 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" path="/var/lib/kubelet/pods/e2f932cd-405e-44fb-87bd-8691fc407440/volumes" Aug 02 10:21:46.549254 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.549221 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx"] Aug 02 10:21:46.549683 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.549639 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" Aug 02 10:21:46.549683 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.549651 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" Aug 02 10:21:46.549683 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.549676 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" Aug 02 10:21:46.549683 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.549681 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" Aug 02 10:21:46.549830 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.549750 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" Aug 02 10:21:46.549830 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.549762 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f932cd-405e-44fb-87bd-8691fc407440" containerName="istio-proxy" Aug 02 10:21:46.552826 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.552810 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.557682 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.557660 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-0c6483-openshift-default-dockercfg-l9r96\"" Aug 02 10:21:46.572960 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.572936 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx"] Aug 02 10:21:46.675415 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.675376 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-token\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.675615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.675423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istiod-ca-cert\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.675615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.675441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5whg\" (UniqueName: \"kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-kube-api-access-h5whg\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.675615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.675497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-credential-socket\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.675615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.675518 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-data\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.675803 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.675618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-podinfo\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.675803 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.675708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-certs\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.675803 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.675767 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-envoy\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.675803 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.675788 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-socket\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.776896 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.776868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-podinfo\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777059 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.776915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-certs\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777059 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.776952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-envoy\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777059 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.776970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-socket\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777059 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.776992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-token\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777279 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.777156 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istiod-ca-cert\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777279 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.777196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5whg\" (UniqueName: \"kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-kube-api-access-h5whg\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777279 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.777239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-credential-socket\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777279 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.777271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-data\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777519 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.777494 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-socket\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777582 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.777524 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-certs\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777732 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.777711 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-credential-socket\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.777786 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.777727 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-data\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.778049 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.778028 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istiod-ca-cert\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.779227 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.779200 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-envoy\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.779682 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.779659 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-podinfo\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.785211 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.785187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-token\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.785363 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.785346 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5whg\" (UniqueName: \"kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-kube-api-access-h5whg\") pod \"e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:46.863581 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:46.863526 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:47.001323 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:47.001295 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx"] Aug 02 10:21:47.003833 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:21:47.003801 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fb8b11_2ba1_4cb0_a80e_d6f240f8cb91.slice/crio-71aebed1f7b00e53786e906697e3d59947a046cf0b12dd525b2de58744640427 WatchSource:0}: Error finding container 71aebed1f7b00e53786e906697e3d59947a046cf0b12dd525b2de58744640427: Status 404 returned error can't find the container with id 71aebed1f7b00e53786e906697e3d59947a046cf0b12dd525b2de58744640427 Aug 02 10:21:47.006029 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:47.005992 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:21:47.006113 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:47.006063 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:21:47.006113 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:47.006092 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:21:47.084830 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:47.084796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" event={"ID":"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91","Type":"ContainerStarted","Data":"abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7"} Aug 02 10:21:47.084951 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:47.084847 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" event={"ID":"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91","Type":"ContainerStarted","Data":"71aebed1f7b00e53786e906697e3d59947a046cf0b12dd525b2de58744640427"} Aug 02 10:21:47.106407 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:47.106352 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" podStartSLOduration=1.106334883 podStartE2EDuration="1.106334883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:21:47.103933455 +0000 UTC m=+2928.544824139" watchObservedRunningTime="2026-08-02 10:21:47.106334883 +0000 UTC m=+2928.547225339" Aug 02 10:21:47.863800 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:47.863763 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:48.868921 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:48.868890 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:49.094690 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:49.094658 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:49.095451 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:49.095429 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:21:53.597540 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:21:53.597504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:22:03.169874 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:03.169778 2564 generic.go:356] "Generic (PLEG): container finished" podID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerID="abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7" exitCode=0 Aug 02 10:22:03.169874 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:03.169851 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" event={"ID":"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91","Type":"ContainerDied","Data":"abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7"} Aug 02 10:22:03.170293 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:03.170228 2564 scope.go:117] "RemoveContainer" containerID="abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7" Aug 02 10:22:03.171184 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:03.171157 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:22:03.171243 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:03.171222 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:22:03.171281 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:03.171254 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:22:04.094953 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:04.094918 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:22:04.176676 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:04.176643 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" event={"ID":"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91","Type":"ContainerStarted","Data":"b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a"} Aug 02 10:22:04.864286 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:04.864243 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:22:04.865551 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:04.865525 2564 patch_prober.go:28] interesting pod/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Aug 02 10:22:04.865689 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:04.865579 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Aug 02 10:22:05.864940 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:05.864899 2564 patch_prober.go:28] interesting pod/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Aug 02 10:22:05.865324 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:05.864957 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Aug 02 10:22:06.864388 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:06.864350 2564 patch_prober.go:28] interesting pod/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Aug 02 10:22:06.864591 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:06.864419 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Aug 02 10:22:07.864623 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:07.864586 2564 patch_prober.go:28] interesting pod/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Aug 02 10:22:07.865008 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:07.864647 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Aug 02 10:22:08.864307 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:08.864275 2564 patch_prober.go:28] interesting pod/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Aug 02 10:22:08.864505 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:08.864337 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Aug 02 10:22:09.867774 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:09.867744 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:22:09.868204 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:09.868022 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:22:09.868653 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:09.868634 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:22:33.802173 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.802094 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq"] Aug 02 10:22:33.804662 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.804643 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:33.807899 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.807879 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"kube-root-ca.crt\"" Aug 02 10:22:33.808017 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.807880 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"auth-test-model-e9225c-kserve-self-signed-certs\"" Aug 02 10:22:33.808017 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.807945 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"openshift-service-ca.crt\"" Aug 02 10:22:33.808017 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.807987 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"default-dockercfg-mxshh\"" Aug 02 10:22:33.816058 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.816033 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq"] Aug 02 10:22:33.919708 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.919667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tls-certs\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:33.919842 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.919731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-dshm\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:33.919842 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.919780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-model-cache\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:33.919842 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.919800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-home\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:33.919842 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.919839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr9hb\" (UniqueName: \"kubernetes.io/projected/41dc89cf-9ebe-47b8-89cf-25d40435d81b-kube-api-access-wr9hb\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:33.919982 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:33.919862 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tmp-dir\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.021359 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.021323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-dshm\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.021359 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.021364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-model-cache\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.021603 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.021512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-home\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.021603 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.021581 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr9hb\" (UniqueName: \"kubernetes.io/projected/41dc89cf-9ebe-47b8-89cf-25d40435d81b-kube-api-access-wr9hb\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.021719 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.021627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tmp-dir\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.021783 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.021717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tls-certs\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.021914 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.021891 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-model-cache\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.021991 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.021935 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-home\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.022051 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.022034 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tmp-dir\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.023718 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.023690 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-dshm\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.024074 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.024058 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tls-certs\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.030070 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.030050 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr9hb\" (UniqueName: \"kubernetes.io/projected/41dc89cf-9ebe-47b8-89cf-25d40435d81b-kube-api-access-wr9hb\") pod \"auth-test-model-e9225c-kserve-7f78d994f9-gmffq\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.115211 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.115135 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:34.249229 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.249203 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq"] Aug 02 10:22:34.251225 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:22:34.251192 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41dc89cf_9ebe_47b8_89cf_25d40435d81b.slice/crio-aef64bc934a4226bec9b4279564b4e8aee445f97dd62feffa68692094e7c923f WatchSource:0}: Error finding container aef64bc934a4226bec9b4279564b4e8aee445f97dd62feffa68692094e7c923f: Status 404 returned error can't find the container with id aef64bc934a4226bec9b4279564b4e8aee445f97dd62feffa68692094e7c923f Aug 02 10:22:34.325814 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.325779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" event={"ID":"41dc89cf-9ebe-47b8-89cf-25d40435d81b","Type":"ContainerStarted","Data":"e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc"} Aug 02 10:22:34.325915 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.325829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" event={"ID":"41dc89cf-9ebe-47b8-89cf-25d40435d81b","Type":"ContainerStarted","Data":"aef64bc934a4226bec9b4279564b4e8aee445f97dd62feffa68692094e7c923f"} Aug 02 10:22:34.347081 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:34.347034 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" podStartSLOduration=1.347003433 podStartE2EDuration="1.347003433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:22:34.346627709 +0000 UTC m=+2975.787518177" watchObservedRunningTime="2026-08-02 10:22:34.347003433 +0000 UTC m=+2975.787893890" Aug 02 10:22:36.388244 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:36.388209 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:22:44.115279 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:44.115241 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:44.115738 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:44.115293 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:44.127630 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:44.127609 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:44.396984 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:44.396908 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:22:46.397328 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:46.397292 2564 generic.go:356] "Generic (PLEG): container finished" podID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerID="3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7" exitCode=0 Aug 02 10:22:46.397770 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:46.397361 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" event={"ID":"10b20eaa-3045-46a2-93c3-1ebefd00ea1d","Type":"ContainerDied","Data":"3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7"} Aug 02 10:22:46.397852 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:46.397837 2564 scope.go:117] "RemoveContainer" containerID="3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7" Aug 02 10:22:46.398854 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:46.398827 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:22:46.398928 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:46.398893 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:22:46.398970 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:46.398930 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:22:47.404455 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:47.404418 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" event={"ID":"10b20eaa-3045-46a2-93c3-1ebefd00ea1d","Type":"ContainerStarted","Data":"be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188"} Aug 02 10:22:47.568543 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:47.568502 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:22:47.569872 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:47.569846 2564 patch_prober.go:28] interesting pod/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Aug 02 10:22:47.569988 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:47.569893 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Aug 02 10:22:48.568785 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:48.568744 2564 patch_prober.go:28] interesting pod/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Aug 02 10:22:48.569176 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:48.568801 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Aug 02 10:22:49.569675 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:49.569636 2564 patch_prober.go:28] interesting pod/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Aug 02 10:22:49.570066 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:49.569695 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Aug 02 10:22:49.948986 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:49.948898 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:22:50.569483 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:50.569433 2564 patch_prober.go:28] interesting pod/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Aug 02 10:22:50.569669 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:50.569513 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Aug 02 10:22:51.569489 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.569426 2564 patch_prober.go:28] interesting pod/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Aug 02 10:22:51.569876 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.569520 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Aug 02 10:22:51.910092 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.910019 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9"] Aug 02 10:22:51.912512 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.912494 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:51.915139 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.915113 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"auth-test-model-0c6483-kserve-self-signed-certs\"" Aug 02 10:22:51.915810 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.915792 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"kube-root-ca.crt\"" Aug 02 10:22:51.915894 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.915820 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"openshift-service-ca.crt\"" Aug 02 10:22:51.915938 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.915888 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"default-dockercfg-wws4r\"" Aug 02 10:22:51.924202 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.924175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9"] Aug 02 10:22:51.994963 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.994919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-model-cache\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:51.995153 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.995000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrzxt\" (UniqueName: \"kubernetes.io/projected/b6204de6-0995-44c7-9d87-3e64b4ce87e5-kube-api-access-xrzxt\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:51.995153 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.995041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-dshm\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:51.995153 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.995134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-home\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:51.995327 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.995190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tls-certs\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:51.995327 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:51.995227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tmp-dir\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.096099 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.096062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-dshm\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.096300 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.096109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-home\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.096300 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.096136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tls-certs\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.096300 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.096155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tmp-dir\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.096300 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.096194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-model-cache\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.096300 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.096249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrzxt\" (UniqueName: \"kubernetes.io/projected/b6204de6-0995-44c7-9d87-3e64b4ce87e5-kube-api-access-xrzxt\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.096679 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.096654 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-home\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.096737 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.096688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tmp-dir\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.096737 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.096711 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-model-cache\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.098425 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.098404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-dshm\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.098748 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.098731 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tls-certs\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.105228 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.105206 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrzxt\" (UniqueName: \"kubernetes.io/projected/b6204de6-0995-44c7-9d87-3e64b4ce87e5-kube-api-access-xrzxt\") pod \"auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.222051 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.222018 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:22:52.397681 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.397637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9"] Aug 02 10:22:52.399973 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:22:52.399947 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6204de6_0995_44c7_9d87_3e64b4ce87e5.slice/crio-3cd8e754ff7d9e4e8da0118ea8f2f2c15cbdc094844644f1246f5f08032a07e6 WatchSource:0}: Error finding container 3cd8e754ff7d9e4e8da0118ea8f2f2c15cbdc094844644f1246f5f08032a07e6: Status 404 returned error can't find the container with id 3cd8e754ff7d9e4e8da0118ea8f2f2c15cbdc094844644f1246f5f08032a07e6 Aug 02 10:22:52.430832 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.430798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" event={"ID":"b6204de6-0995-44c7-9d87-3e64b4ce87e5","Type":"ContainerStarted","Data":"3cd8e754ff7d9e4e8da0118ea8f2f2c15cbdc094844644f1246f5f08032a07e6"} Aug 02 10:22:52.569100 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.569061 2564 patch_prober.go:28] interesting pod/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Aug 02 10:22:52.569306 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:52.569116 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Aug 02 10:22:53.437497 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:53.437443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" event={"ID":"b6204de6-0995-44c7-9d87-3e64b4ce87e5","Type":"ContainerStarted","Data":"d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837"} Aug 02 10:22:53.458448 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:53.458403 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" podStartSLOduration=2.458389151 podStartE2EDuration="2.458389151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:22:53.456497181 +0000 UTC m=+2994.897387638" watchObservedRunningTime="2026-08-02 10:22:53.458389151 +0000 UTC m=+2994.899279606" Aug 02 10:22:53.572882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:53.572851 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:22:53.573846 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:53.573823 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:22:54.495555 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:22:54.495521 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:23:02.222556 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:23:02.222523 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:23:02.222953 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:23:02.222782 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:23:02.234968 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:23:02.234942 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:23:02.493107 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:23:02.493026 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:23:19.600350 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:23:19.600314 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:23:35.191856 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:23:35.191822 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:23:46.534914 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:23:46.534879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:24:01.701689 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:24:01.701602 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:24:17.123169 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:24:17.123138 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:24:27.798058 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:24:27.798028 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:24:43.489284 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:24:43.489249 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:25:09.298770 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:25:09.298740 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:25:20.005647 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:25:20.005608 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:25:35.890880 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:25:35.890794 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:25:50.994627 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:25:50.994598 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:26:01.901400 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:26:01.901365 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:26:17.492081 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:26:17.492043 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:26:33.196890 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:26:33.196852 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:26:44.191695 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:26:44.191653 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:26:59.492544 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:26:59.492471 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:27:15.189902 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:15.189865 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:27:18.994845 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:18.994804 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq"] Aug 02 10:27:19.901485 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:19.901434 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9"] Aug 02 10:27:26.099172 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:26.099143 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:27:32.002699 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:32.002657 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:27:34.014107 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.014050 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" podUID="41dc89cf-9ebe-47b8-89cf-25d40435d81b" containerName="main" containerID="cri-o://e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc" gracePeriod=45 Aug 02 10:27:34.262148 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.262126 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:27:34.377852 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.377774 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-dshm\") pod \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " Aug 02 10:27:34.377852 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.377820 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-model-cache\") pod \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " Aug 02 10:27:34.377852 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.377840 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tls-certs\") pod \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " Aug 02 10:27:34.378116 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.377873 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-home\") pod \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " Aug 02 10:27:34.378116 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.377937 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr9hb\" (UniqueName: \"kubernetes.io/projected/41dc89cf-9ebe-47b8-89cf-25d40435d81b-kube-api-access-wr9hb\") pod \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " Aug 02 10:27:34.378116 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.378001 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tmp-dir\") pod \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\" (UID: \"41dc89cf-9ebe-47b8-89cf-25d40435d81b\") " Aug 02 10:27:34.378116 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.378087 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-model-cache" (OuterVolumeSpecName: "model-cache") pod "41dc89cf-9ebe-47b8-89cf-25d40435d81b" (UID: "41dc89cf-9ebe-47b8-89cf-25d40435d81b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:27:34.378318 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.378160 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-home" (OuterVolumeSpecName: "home") pod "41dc89cf-9ebe-47b8-89cf-25d40435d81b" (UID: "41dc89cf-9ebe-47b8-89cf-25d40435d81b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:27:34.378318 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.378294 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41dc89cf-9ebe-47b8-89cf-25d40435d81b" (UID: "41dc89cf-9ebe-47b8-89cf-25d40435d81b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:27:34.378423 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.378360 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:34.378423 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.378373 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:34.378423 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.378382 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:34.380078 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.380050 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-dshm" (OuterVolumeSpecName: "dshm") pod "41dc89cf-9ebe-47b8-89cf-25d40435d81b" (UID: "41dc89cf-9ebe-47b8-89cf-25d40435d81b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:27:34.380186 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.380122 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41dc89cf-9ebe-47b8-89cf-25d40435d81b" (UID: "41dc89cf-9ebe-47b8-89cf-25d40435d81b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 10:27:34.380186 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.380131 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41dc89cf-9ebe-47b8-89cf-25d40435d81b-kube-api-access-wr9hb" (OuterVolumeSpecName: "kube-api-access-wr9hb") pod "41dc89cf-9ebe-47b8-89cf-25d40435d81b" (UID: "41dc89cf-9ebe-47b8-89cf-25d40435d81b"). InnerVolumeSpecName "kube-api-access-wr9hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:27:34.479354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.479318 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr9hb\" (UniqueName: \"kubernetes.io/projected/41dc89cf-9ebe-47b8-89cf-25d40435d81b-kube-api-access-wr9hb\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:34.479354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.479353 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41dc89cf-9ebe-47b8-89cf-25d40435d81b-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:34.479354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.479363 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41dc89cf-9ebe-47b8-89cf-25d40435d81b-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:34.843532 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.843495 2564 generic.go:356] "Generic (PLEG): container finished" podID="41dc89cf-9ebe-47b8-89cf-25d40435d81b" containerID="e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc" exitCode=0 Aug 02 10:27:34.843707 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.843596 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" Aug 02 10:27:34.843707 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.843622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" event={"ID":"41dc89cf-9ebe-47b8-89cf-25d40435d81b","Type":"ContainerDied","Data":"e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc"} Aug 02 10:27:34.843707 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.843653 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq" event={"ID":"41dc89cf-9ebe-47b8-89cf-25d40435d81b","Type":"ContainerDied","Data":"aef64bc934a4226bec9b4279564b4e8aee445f97dd62feffa68692094e7c923f"} Aug 02 10:27:34.843707 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.843670 2564 scope.go:117] "RemoveContainer" containerID="e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc" Aug 02 10:27:34.852788 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.852769 2564 scope.go:117] "RemoveContainer" containerID="e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc" Aug 02 10:27:34.853044 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:27:34.853028 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc\": container with ID starting with e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc not found: ID does not exist" containerID="e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc" Aug 02 10:27:34.853102 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.853054 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc"} err="failed to get container status \"e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc\": rpc error: code = NotFound desc = could not find container \"e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc\": container with ID starting with e098ca00522d744d4a9cdb66b1c0e66d89c778c2e3b966cef7aceeb54bf99ecc not found: ID does not exist" Aug 02 10:27:34.866066 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.866046 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq"] Aug 02 10:27:34.873726 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.873692 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/auth-test-model-e9225c-kserve-7f78d994f9-gmffq"] Aug 02 10:27:34.920510 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.920482 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" podUID="b6204de6-0995-44c7-9d87-3e64b4ce87e5" containerName="main" containerID="cri-o://d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837" gracePeriod=45 Aug 02 10:27:34.978355 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:34.978328 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41dc89cf-9ebe-47b8-89cf-25d40435d81b" path="/var/lib/kubelet/pods/41dc89cf-9ebe-47b8-89cf-25d40435d81b/volumes" Aug 02 10:27:35.164028 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.164002 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:27:35.186058 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186036 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-model-cache\") pod \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " Aug 02 10:27:35.186416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186064 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-home\") pod \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " Aug 02 10:27:35.186416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186095 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrzxt\" (UniqueName: \"kubernetes.io/projected/b6204de6-0995-44c7-9d87-3e64b4ce87e5-kube-api-access-xrzxt\") pod \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " Aug 02 10:27:35.186416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186120 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tmp-dir\") pod \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " Aug 02 10:27:35.186416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186299 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tls-certs\") pod \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " Aug 02 10:27:35.186416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186359 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6204de6-0995-44c7-9d87-3e64b4ce87e5" (UID: "b6204de6-0995-44c7-9d87-3e64b4ce87e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:27:35.186416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186368 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-dshm\") pod \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\" (UID: \"b6204de6-0995-44c7-9d87-3e64b4ce87e5\") " Aug 02 10:27:35.186783 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186376 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-home" (OuterVolumeSpecName: "home") pod "b6204de6-0995-44c7-9d87-3e64b4ce87e5" (UID: "b6204de6-0995-44c7-9d87-3e64b4ce87e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:27:35.186783 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186386 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6204de6-0995-44c7-9d87-3e64b4ce87e5" (UID: "b6204de6-0995-44c7-9d87-3e64b4ce87e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:27:35.186783 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186748 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:35.186783 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186766 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:35.186783 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.186778 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:35.188396 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.188376 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6204de6-0995-44c7-9d87-3e64b4ce87e5" (UID: "b6204de6-0995-44c7-9d87-3e64b4ce87e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 10:27:35.188516 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.188434 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6204de6-0995-44c7-9d87-3e64b4ce87e5-kube-api-access-xrzxt" (OuterVolumeSpecName: "kube-api-access-xrzxt") pod "b6204de6-0995-44c7-9d87-3e64b4ce87e5" (UID: "b6204de6-0995-44c7-9d87-3e64b4ce87e5"). InnerVolumeSpecName "kube-api-access-xrzxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:27:35.188700 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.188685 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-dshm" (OuterVolumeSpecName: "dshm") pod "b6204de6-0995-44c7-9d87-3e64b4ce87e5" (UID: "b6204de6-0995-44c7-9d87-3e64b4ce87e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:27:35.287315 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.287289 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrzxt\" (UniqueName: \"kubernetes.io/projected/b6204de6-0995-44c7-9d87-3e64b4ce87e5-kube-api-access-xrzxt\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:35.287315 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.287312 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6204de6-0995-44c7-9d87-3e64b4ce87e5-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:35.287453 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.287321 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6204de6-0995-44c7-9d87-3e64b4ce87e5-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:27:35.850205 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.850172 2564 generic.go:356] "Generic (PLEG): container finished" podID="b6204de6-0995-44c7-9d87-3e64b4ce87e5" containerID="d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837" exitCode=0 Aug 02 10:27:35.850396 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.850221 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" Aug 02 10:27:35.850396 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.850265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" event={"ID":"b6204de6-0995-44c7-9d87-3e64b4ce87e5","Type":"ContainerDied","Data":"d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837"} Aug 02 10:27:35.850396 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.850313 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9" event={"ID":"b6204de6-0995-44c7-9d87-3e64b4ce87e5","Type":"ContainerDied","Data":"3cd8e754ff7d9e4e8da0118ea8f2f2c15cbdc094844644f1246f5f08032a07e6"} Aug 02 10:27:35.850396 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.850336 2564 scope.go:117] "RemoveContainer" containerID="d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837" Aug 02 10:27:35.860146 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.860129 2564 scope.go:117] "RemoveContainer" containerID="d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837" Aug 02 10:27:35.860374 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:27:35.860357 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837\": container with ID starting with d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837 not found: ID does not exist" containerID="d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837" Aug 02 10:27:35.860426 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.860382 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837"} err="failed to get container status \"d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837\": rpc error: code = NotFound desc = could not find container \"d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837\": container with ID starting with d4ae4eb0d66c20cf4d391cada16f1428eb85b83aa62e1d6e5b6ff736b992f837 not found: ID does not exist" Aug 02 10:27:35.872420 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.872400 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9"] Aug 02 10:27:35.875479 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:35.875438 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/auth-test-model-0c6483-kserve-7f89b47f9f-xdbd9"] Aug 02 10:27:36.978707 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:27:36.978673 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6204de6-0995-44c7-9d87-3e64b4ce87e5" path="/var/lib/kubelet/pods/b6204de6-0995-44c7-9d87-3e64b4ce87e5/volumes" Aug 02 10:28:28.891258 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.891175 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk"] Aug 02 10:28:28.891792 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.891773 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41dc89cf-9ebe-47b8-89cf-25d40435d81b" containerName="main" Aug 02 10:28:28.891874 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.891797 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="41dc89cf-9ebe-47b8-89cf-25d40435d81b" containerName="main" Aug 02 10:28:28.891874 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.891811 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6204de6-0995-44c7-9d87-3e64b4ce87e5" containerName="main" Aug 02 10:28:28.891874 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.891821 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6204de6-0995-44c7-9d87-3e64b4ce87e5" containerName="main" Aug 02 10:28:28.892024 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.891929 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="41dc89cf-9ebe-47b8-89cf-25d40435d81b" containerName="main" Aug 02 10:28:28.892024 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.891944 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6204de6-0995-44c7-9d87-3e64b4ce87e5" containerName="main" Aug 02 10:28:28.895089 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.895069 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:28.899053 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.899029 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"default-dockercfg-wws4r\"" Aug 02 10:28:28.899173 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.899048 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"kube-root-ca.crt\"" Aug 02 10:28:28.899173 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.899029 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"sub-test-model-0c6483-kserve-self-signed-certs\"" Aug 02 10:28:28.899173 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.899143 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"openshift-service-ca.crt\"" Aug 02 10:28:28.904287 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.904258 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk"] Aug 02 10:28:28.957408 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.957380 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldsjw\" (UniqueName: \"kubernetes.io/projected/8231f3b9-548e-4b03-95e4-3186bbde3665-kube-api-access-ldsjw\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:28.957548 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.957436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-tmp-dir\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:28.957548 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.957501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8231f3b9-548e-4b03-95e4-3186bbde3665-tls-certs\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:28.957633 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.957548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-home\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:28.957633 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.957565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-dshm\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:28.957633 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:28.957613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-model-cache\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.058897 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.058864 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldsjw\" (UniqueName: \"kubernetes.io/projected/8231f3b9-548e-4b03-95e4-3186bbde3665-kube-api-access-ldsjw\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.059085 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.058972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-tmp-dir\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.059085 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.059009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8231f3b9-548e-4b03-95e4-3186bbde3665-tls-certs\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.059085 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.059067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-home\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.059272 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.059113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-dshm\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.059272 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.059153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-model-cache\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.059454 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.059426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-tmp-dir\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.059563 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.059512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-model-cache\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.059563 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.059549 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-home\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.061634 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.061607 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-dshm\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.061942 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.061925 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8231f3b9-548e-4b03-95e4-3186bbde3665-tls-certs\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.066671 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.066650 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldsjw\" (UniqueName: \"kubernetes.io/projected/8231f3b9-548e-4b03-95e4-3186bbde3665-kube-api-access-ldsjw\") pod \"sub-test-model-0c6483-kserve-76b548d9f5-smdnk\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.207692 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.207668 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:29.339514 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:29.339488 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk"] Aug 02 10:28:29.341762 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:28:29.341736 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8231f3b9_548e_4b03_95e4_3186bbde3665.slice/crio-0e04452ff74a88702ed54ada1ee65d5820eff5d5381d6f33b857ea7a710c546a WatchSource:0}: Error finding container 0e04452ff74a88702ed54ada1ee65d5820eff5d5381d6f33b857ea7a710c546a: Status 404 returned error can't find the container with id 0e04452ff74a88702ed54ada1ee65d5820eff5d5381d6f33b857ea7a710c546a Aug 02 10:28:30.126307 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:30.126273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" event={"ID":"8231f3b9-548e-4b03-95e4-3186bbde3665","Type":"ContainerStarted","Data":"f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f"} Aug 02 10:28:30.126780 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:30.126316 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" event={"ID":"8231f3b9-548e-4b03-95e4-3186bbde3665","Type":"ContainerStarted","Data":"0e04452ff74a88702ed54ada1ee65d5820eff5d5381d6f33b857ea7a710c546a"} Aug 02 10:28:30.143625 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:30.143568 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" podStartSLOduration=2.143552441 podStartE2EDuration="2.143552441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:28:30.142602013 +0000 UTC m=+3331.583492471" watchObservedRunningTime="2026-08-02 10:28:30.143552441 +0000 UTC m=+3331.584442959" Aug 02 10:28:31.701609 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:31.701578 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:28:39.208259 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:39.208222 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:39.208750 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:39.208277 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:39.220054 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:39.220023 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:28:40.186403 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:28:40.186368 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:29:08.284583 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:08.284546 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:29:21.899457 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:21.899413 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:29:33.286897 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:33.286848 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:29:39.680353 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:39.680323 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk"] Aug 02 10:29:41.590388 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:41.590351 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:29:53.104554 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.104457 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr"] Aug 02 10:29:53.108120 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.108096 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.111319 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.111295 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"openshift-service-ca.crt\"" Aug 02 10:29:53.111438 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.111361 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"kube-root-ca.crt\"" Aug 02 10:29:53.111438 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.111361 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"rate-test-model-e9225c-kserve-self-signed-certs\"" Aug 02 10:29:53.113510 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.113451 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e9225c\"/\"default-dockercfg-mxshh\"" Aug 02 10:29:53.127534 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.127481 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr"] Aug 02 10:29:53.142537 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.142507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-tmp-dir\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.142673 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.142582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63a47e6-0856-4a00-b308-c6ef67168d37-tls-certs\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.142673 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.142646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-model-cache\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.142771 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.142685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-home\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.142771 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.142732 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-dshm\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.142771 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.142754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5v9\" (UniqueName: \"kubernetes.io/projected/c63a47e6-0856-4a00-b308-c6ef67168d37-kube-api-access-zg5v9\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.244310 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.244273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-home\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.244497 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.244349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-dshm\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.244497 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.244376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5v9\" (UniqueName: \"kubernetes.io/projected/c63a47e6-0856-4a00-b308-c6ef67168d37-kube-api-access-zg5v9\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.244497 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.244445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-tmp-dir\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.244690 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.244553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63a47e6-0856-4a00-b308-c6ef67168d37-tls-certs\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.244752 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.244737 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-home\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.245186 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.245163 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-model-cache\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.245379 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.245233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-model-cache\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.245574 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.245243 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-tmp-dir\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.247038 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.247016 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-dshm\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.247418 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.247401 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63a47e6-0856-4a00-b308-c6ef67168d37-tls-certs\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.257053 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.257031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5v9\" (UniqueName: \"kubernetes.io/projected/c63a47e6-0856-4a00-b308-c6ef67168d37-kube-api-access-zg5v9\") pod \"rate-test-model-e9225c-kserve-86c9478459-jgcrr\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.418014 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.417951 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:29:53.552350 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:53.552326 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr"] Aug 02 10:29:53.554349 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:29:53.554323 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc63a47e6_0856_4a00_b308_c6ef67168d37.slice/crio-a54f8e8fd7439c9c97cb5a793020d4524ea374c8b399dd01f2005db061740ca2 WatchSource:0}: Error finding container a54f8e8fd7439c9c97cb5a793020d4524ea374c8b399dd01f2005db061740ca2: Status 404 returned error can't find the container with id a54f8e8fd7439c9c97cb5a793020d4524ea374c8b399dd01f2005db061740ca2 Aug 02 10:29:54.537959 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:54.537926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" event={"ID":"c63a47e6-0856-4a00-b308-c6ef67168d37","Type":"ContainerStarted","Data":"04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25"} Aug 02 10:29:54.538334 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:54.537967 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" event={"ID":"c63a47e6-0856-4a00-b308-c6ef67168d37","Type":"ContainerStarted","Data":"a54f8e8fd7439c9c97cb5a793020d4524ea374c8b399dd01f2005db061740ca2"} Aug 02 10:29:54.557028 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:54.556982 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" podStartSLOduration=1.556966638 podStartE2EDuration="1.556966638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:29:54.555140897 +0000 UTC m=+3415.996031353" watchObservedRunningTime="2026-08-02 10:29:54.556966638 +0000 UTC m=+3415.997857094" Aug 02 10:29:54.699330 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:54.699291 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" podUID="8231f3b9-548e-4b03-95e4-3186bbde3665" containerName="main" containerID="cri-o://f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f" gracePeriod=45 Aug 02 10:29:54.942664 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:54.942641 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:29:55.063719 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.063679 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-dshm\") pod \"8231f3b9-548e-4b03-95e4-3186bbde3665\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " Aug 02 10:29:55.063719 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.063725 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-model-cache\") pod \"8231f3b9-548e-4b03-95e4-3186bbde3665\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " Aug 02 10:29:55.063962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.063811 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-tmp-dir\") pod \"8231f3b9-548e-4b03-95e4-3186bbde3665\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " Aug 02 10:29:55.063962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.063930 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8231f3b9-548e-4b03-95e4-3186bbde3665-tls-certs\") pod \"8231f3b9-548e-4b03-95e4-3186bbde3665\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " Aug 02 10:29:55.064074 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.063962 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldsjw\" (UniqueName: \"kubernetes.io/projected/8231f3b9-548e-4b03-95e4-3186bbde3665-kube-api-access-ldsjw\") pod \"8231f3b9-548e-4b03-95e4-3186bbde3665\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " Aug 02 10:29:55.064074 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.063984 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-home\") pod \"8231f3b9-548e-4b03-95e4-3186bbde3665\" (UID: \"8231f3b9-548e-4b03-95e4-3186bbde3665\") " Aug 02 10:29:55.064074 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.064012 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-model-cache" (OuterVolumeSpecName: "model-cache") pod "8231f3b9-548e-4b03-95e4-3186bbde3665" (UID: "8231f3b9-548e-4b03-95e4-3186bbde3665"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:29:55.064290 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.064160 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8231f3b9-548e-4b03-95e4-3186bbde3665" (UID: "8231f3b9-548e-4b03-95e4-3186bbde3665"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:29:55.064375 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.064352 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-home" (OuterVolumeSpecName: "home") pod "8231f3b9-548e-4b03-95e4-3186bbde3665" (UID: "8231f3b9-548e-4b03-95e4-3186bbde3665"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:29:55.064437 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.064383 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:29:55.064437 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.064403 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:29:55.066108 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.066037 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8231f3b9-548e-4b03-95e4-3186bbde3665-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8231f3b9-548e-4b03-95e4-3186bbde3665" (UID: "8231f3b9-548e-4b03-95e4-3186bbde3665"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 10:29:55.066108 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.066044 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8231f3b9-548e-4b03-95e4-3186bbde3665-kube-api-access-ldsjw" (OuterVolumeSpecName: "kube-api-access-ldsjw") pod "8231f3b9-548e-4b03-95e4-3186bbde3665" (UID: "8231f3b9-548e-4b03-95e4-3186bbde3665"). InnerVolumeSpecName "kube-api-access-ldsjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:29:55.066296 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.066124 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-dshm" (OuterVolumeSpecName: "dshm") pod "8231f3b9-548e-4b03-95e4-3186bbde3665" (UID: "8231f3b9-548e-4b03-95e4-3186bbde3665"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:29:55.165416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.165383 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8231f3b9-548e-4b03-95e4-3186bbde3665-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:29:55.165416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.165412 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldsjw\" (UniqueName: \"kubernetes.io/projected/8231f3b9-548e-4b03-95e4-3186bbde3665-kube-api-access-ldsjw\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:29:55.165416 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.165421 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:29:55.165655 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.165430 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8231f3b9-548e-4b03-95e4-3186bbde3665-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:29:55.545438 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.545402 2564 generic.go:356] "Generic (PLEG): container finished" podID="8231f3b9-548e-4b03-95e4-3186bbde3665" containerID="f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f" exitCode=0 Aug 02 10:29:55.545917 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.545482 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" Aug 02 10:29:55.545917 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.545494 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" event={"ID":"8231f3b9-548e-4b03-95e4-3186bbde3665","Type":"ContainerDied","Data":"f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f"} Aug 02 10:29:55.545917 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.545545 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk" event={"ID":"8231f3b9-548e-4b03-95e4-3186bbde3665","Type":"ContainerDied","Data":"0e04452ff74a88702ed54ada1ee65d5820eff5d5381d6f33b857ea7a710c546a"} Aug 02 10:29:55.545917 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.545568 2564 scope.go:117] "RemoveContainer" containerID="f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f" Aug 02 10:29:55.554862 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.554845 2564 scope.go:117] "RemoveContainer" containerID="f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f" Aug 02 10:29:55.555112 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:29:55.555091 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f\": container with ID starting with f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f not found: ID does not exist" containerID="f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f" Aug 02 10:29:55.555166 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.555120 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f"} err="failed to get container status \"f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f\": rpc error: code = NotFound desc = could not find container \"f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f\": container with ID starting with f0f9d2f95dc3994bd718525d0844179ebda55245058015c905b9a988c1a8a91f not found: ID does not exist" Aug 02 10:29:55.568141 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.568112 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk"] Aug 02 10:29:55.572440 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.572419 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/sub-test-model-0c6483-kserve-76b548d9f5-smdnk"] Aug 02 10:29:55.894746 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:55.894664 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:29:56.978652 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:29:56.978616 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8231f3b9-548e-4b03-95e4-3186bbde3665" path="/var/lib/kubelet/pods/8231f3b9-548e-4b03-95e4-3186bbde3665/volumes" Aug 02 10:30:03.418826 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:03.418791 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:30:03.419220 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:03.418840 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:30:03.431230 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:03.431202 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:30:03.595499 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:03.595473 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:30:29.102033 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.101998 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j"] Aug 02 10:30:29.102418 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.102408 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8231f3b9-548e-4b03-95e4-3186bbde3665" containerName="main" Aug 02 10:30:29.102418 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.102418 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8231f3b9-548e-4b03-95e4-3186bbde3665" containerName="main" Aug 02 10:30:29.102520 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.102489 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8231f3b9-548e-4b03-95e4-3186bbde3665" containerName="main" Aug 02 10:30:29.105408 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.105392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.107744 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.107717 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"default-dockercfg-wws4r\"" Aug 02 10:30:29.108671 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.108652 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"openshift-service-ca.crt\"" Aug 02 10:30:29.108749 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.108672 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"kube-root-ca.crt\"" Aug 02 10:30:29.108749 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.108681 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c6483\"/\"rate-test-model-0c6483-kserve-self-signed-certs\"" Aug 02 10:30:29.114198 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.114170 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j"] Aug 02 10:30:29.177869 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.177829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-dshm\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.178088 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.177915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tmp-dir\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.178088 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.177967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-model-cache\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.178088 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.178012 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-home\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.178088 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.178085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tls-certs\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.178317 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.178121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rztk\" (UniqueName: \"kubernetes.io/projected/964c8dfc-56e0-443f-9c2d-0c2642fdea77-kube-api-access-6rztk\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.278824 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.278740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-dshm\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.278824 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.278770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tmp-dir\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.278824 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.278802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-model-cache\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.279103 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.278833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-home\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.279103 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.278874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tls-certs\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.279103 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.279019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rztk\" (UniqueName: \"kubernetes.io/projected/964c8dfc-56e0-443f-9c2d-0c2642fdea77-kube-api-access-6rztk\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.279258 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.279137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tmp-dir\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.279258 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.279211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-home\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.279387 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.279367 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-model-cache\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.281648 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.281615 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-dshm\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.281863 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.281836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tls-certs\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.286927 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.286905 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rztk\" (UniqueName: \"kubernetes.io/projected/964c8dfc-56e0-443f-9c2d-0c2642fdea77-kube-api-access-6rztk\") pod \"rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.415943 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.415910 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:29.546319 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.546231 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j"] Aug 02 10:30:29.550652 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:30:29.550614 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod964c8dfc_56e0_443f_9c2d_0c2642fdea77.slice/crio-89e513f38f3d72f902f3c3ff18c867114728b614003b778e618a34900c173e59 WatchSource:0}: Error finding container 89e513f38f3d72f902f3c3ff18c867114728b614003b778e618a34900c173e59: Status 404 returned error can't find the container with id 89e513f38f3d72f902f3c3ff18c867114728b614003b778e618a34900c173e59 Aug 02 10:30:29.714708 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.714666 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" event={"ID":"964c8dfc-56e0-443f-9c2d-0c2642fdea77","Type":"ContainerStarted","Data":"c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8"} Aug 02 10:30:29.714876 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:29.714720 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" event={"ID":"964c8dfc-56e0-443f-9c2d-0c2642fdea77","Type":"ContainerStarted","Data":"89e513f38f3d72f902f3c3ff18c867114728b614003b778e618a34900c173e59"} Aug 02 10:30:31.708347 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:31.708295 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" podStartSLOduration=2.70827861 podStartE2EDuration="2.70827861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:30:29.733500281 +0000 UTC m=+3451.174390738" watchObservedRunningTime="2026-08-02 10:30:31.70827861 +0000 UTC m=+3453.149169066" Aug 02 10:30:31.709484 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:31.709447 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:30:39.416786 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:39.416752 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:39.416786 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:39.416793 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:39.429139 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:39.429113 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:30:39.775646 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:30:39.775612 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:31:07.800962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:07.800930 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:31:15.294986 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:15.294953 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:31:32.108658 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:32.108621 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:31:37.775124 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:37.775089 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr"] Aug 02 10:31:38.840749 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:38.840713 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j"] Aug 02 10:31:41.832381 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:41.832348 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:31:52.793781 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:52.793743 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" podUID="c63a47e6-0856-4a00-b308-c6ef67168d37" containerName="main" containerID="cri-o://04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25" gracePeriod=45 Aug 02 10:31:53.047660 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.047605 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:31:53.115509 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.115454 2564 generic.go:356] "Generic (PLEG): container finished" podID="c63a47e6-0856-4a00-b308-c6ef67168d37" containerID="04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25" exitCode=0 Aug 02 10:31:53.115509 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.115495 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" event={"ID":"c63a47e6-0856-4a00-b308-c6ef67168d37","Type":"ContainerDied","Data":"04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25"} Aug 02 10:31:53.115509 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.115513 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" Aug 02 10:31:53.115727 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.115548 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr" event={"ID":"c63a47e6-0856-4a00-b308-c6ef67168d37","Type":"ContainerDied","Data":"a54f8e8fd7439c9c97cb5a793020d4524ea374c8b399dd01f2005db061740ca2"} Aug 02 10:31:53.115727 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.115572 2564 scope.go:117] "RemoveContainer" containerID="04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25" Aug 02 10:31:53.123961 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.123945 2564 scope.go:117] "RemoveContainer" containerID="04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25" Aug 02 10:31:53.124204 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:31:53.124186 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25\": container with ID starting with 04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25 not found: ID does not exist" containerID="04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25" Aug 02 10:31:53.124251 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.124214 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25"} err="failed to get container status \"04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25\": rpc error: code = NotFound desc = could not find container \"04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25\": container with ID starting with 04578f6d9e464f42a03f303330f4cca65dec031a42b90e2c290857a591fffe25 not found: ID does not exist" Aug 02 10:31:53.129596 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.129579 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-tmp-dir\") pod \"c63a47e6-0856-4a00-b308-c6ef67168d37\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " Aug 02 10:31:53.129678 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.129625 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-dshm\") pod \"c63a47e6-0856-4a00-b308-c6ef67168d37\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " Aug 02 10:31:53.129678 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.129670 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-model-cache\") pod \"c63a47e6-0856-4a00-b308-c6ef67168d37\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " Aug 02 10:31:53.129786 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.129704 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg5v9\" (UniqueName: \"kubernetes.io/projected/c63a47e6-0856-4a00-b308-c6ef67168d37-kube-api-access-zg5v9\") pod \"c63a47e6-0856-4a00-b308-c6ef67168d37\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " Aug 02 10:31:53.129786 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.129728 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63a47e6-0856-4a00-b308-c6ef67168d37-tls-certs\") pod \"c63a47e6-0856-4a00-b308-c6ef67168d37\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " Aug 02 10:31:53.129786 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.129742 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-home\") pod \"c63a47e6-0856-4a00-b308-c6ef67168d37\" (UID: \"c63a47e6-0856-4a00-b308-c6ef67168d37\") " Aug 02 10:31:53.129940 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.129868 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c63a47e6-0856-4a00-b308-c6ef67168d37" (UID: "c63a47e6-0856-4a00-b308-c6ef67168d37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:31:53.130045 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.130019 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-model-cache" (OuterVolumeSpecName: "model-cache") pod "c63a47e6-0856-4a00-b308-c6ef67168d37" (UID: "c63a47e6-0856-4a00-b308-c6ef67168d37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:31:53.130107 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.130046 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:53.130107 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.130057 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-home" (OuterVolumeSpecName: "home") pod "c63a47e6-0856-4a00-b308-c6ef67168d37" (UID: "c63a47e6-0856-4a00-b308-c6ef67168d37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:31:53.131627 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.131602 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-dshm" (OuterVolumeSpecName: "dshm") pod "c63a47e6-0856-4a00-b308-c6ef67168d37" (UID: "c63a47e6-0856-4a00-b308-c6ef67168d37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:31:53.131784 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.131764 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63a47e6-0856-4a00-b308-c6ef67168d37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c63a47e6-0856-4a00-b308-c6ef67168d37" (UID: "c63a47e6-0856-4a00-b308-c6ef67168d37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 10:31:53.131832 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.131792 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63a47e6-0856-4a00-b308-c6ef67168d37-kube-api-access-zg5v9" (OuterVolumeSpecName: "kube-api-access-zg5v9") pod "c63a47e6-0856-4a00-b308-c6ef67168d37" (UID: "c63a47e6-0856-4a00-b308-c6ef67168d37"). InnerVolumeSpecName "kube-api-access-zg5v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:31:53.231273 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.231242 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:53.231273 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.231270 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:53.231450 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.231281 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg5v9\" (UniqueName: \"kubernetes.io/projected/c63a47e6-0856-4a00-b308-c6ef67168d37-kube-api-access-zg5v9\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:53.231450 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.231289 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63a47e6-0856-4a00-b308-c6ef67168d37-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:53.231450 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.231298 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63a47e6-0856-4a00-b308-c6ef67168d37-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:53.447126 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.447096 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr"] Aug 02 10:31:53.450392 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.450369 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e9225c/rate-test-model-e9225c-kserve-86c9478459-jgcrr"] Aug 02 10:31:53.859857 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:53.859789 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" podUID="964c8dfc-56e0-443f-9c2d-0c2642fdea77" containerName="main" containerID="cri-o://c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8" gracePeriod=45 Aug 02 10:31:54.108181 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.108157 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:31:54.122681 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.122616 2564 generic.go:356] "Generic (PLEG): container finished" podID="964c8dfc-56e0-443f-9c2d-0c2642fdea77" containerID="c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8" exitCode=0 Aug 02 10:31:54.122681 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.122649 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" Aug 02 10:31:54.122835 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.122702 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" event={"ID":"964c8dfc-56e0-443f-9c2d-0c2642fdea77","Type":"ContainerDied","Data":"c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8"} Aug 02 10:31:54.122835 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.122738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j" event={"ID":"964c8dfc-56e0-443f-9c2d-0c2642fdea77","Type":"ContainerDied","Data":"89e513f38f3d72f902f3c3ff18c867114728b614003b778e618a34900c173e59"} Aug 02 10:31:54.122835 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.122756 2564 scope.go:117] "RemoveContainer" containerID="c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8" Aug 02 10:31:54.131027 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.131006 2564 scope.go:117] "RemoveContainer" containerID="c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8" Aug 02 10:31:54.131293 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:31:54.131276 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8\": container with ID starting with c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8 not found: ID does not exist" containerID="c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8" Aug 02 10:31:54.131360 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.131306 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8"} err="failed to get container status \"c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8\": rpc error: code = NotFound desc = could not find container \"c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8\": container with ID starting with c7d474c7db12138d042bbb622fe00dea56e51ccfa176c2c073ecb0fc21dfb8c8 not found: ID does not exist" Aug 02 10:31:54.138769 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.138738 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-dshm\") pod \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " Aug 02 10:31:54.138884 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.138829 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tls-certs\") pod \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " Aug 02 10:31:54.138957 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.138881 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-home\") pod \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " Aug 02 10:31:54.138957 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.138949 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tmp-dir\") pod \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " Aug 02 10:31:54.139072 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.138995 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-model-cache\") pod \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " Aug 02 10:31:54.139072 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.139022 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rztk\" (UniqueName: \"kubernetes.io/projected/964c8dfc-56e0-443f-9c2d-0c2642fdea77-kube-api-access-6rztk\") pod \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\" (UID: \"964c8dfc-56e0-443f-9c2d-0c2642fdea77\") " Aug 02 10:31:54.139424 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.139367 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-home" (OuterVolumeSpecName: "home") pod "964c8dfc-56e0-443f-9c2d-0c2642fdea77" (UID: "964c8dfc-56e0-443f-9c2d-0c2642fdea77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:31:54.139754 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.139694 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "964c8dfc-56e0-443f-9c2d-0c2642fdea77" (UID: "964c8dfc-56e0-443f-9c2d-0c2642fdea77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:31:54.140601 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.140573 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-model-cache" (OuterVolumeSpecName: "model-cache") pod "964c8dfc-56e0-443f-9c2d-0c2642fdea77" (UID: "964c8dfc-56e0-443f-9c2d-0c2642fdea77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:31:54.141733 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.141707 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-dshm" (OuterVolumeSpecName: "dshm") pod "964c8dfc-56e0-443f-9c2d-0c2642fdea77" (UID: "964c8dfc-56e0-443f-9c2d-0c2642fdea77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:31:54.141940 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.141916 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "964c8dfc-56e0-443f-9c2d-0c2642fdea77" (UID: "964c8dfc-56e0-443f-9c2d-0c2642fdea77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 10:31:54.141940 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.141921 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964c8dfc-56e0-443f-9c2d-0c2642fdea77-kube-api-access-6rztk" (OuterVolumeSpecName: "kube-api-access-6rztk") pod "964c8dfc-56e0-443f-9c2d-0c2642fdea77" (UID: "964c8dfc-56e0-443f-9c2d-0c2642fdea77"). InnerVolumeSpecName "kube-api-access-6rztk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:31:54.240598 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.240568 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:54.240747 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.240618 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:54.240747 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.240632 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:54.240747 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.240643 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rztk\" (UniqueName: \"kubernetes.io/projected/964c8dfc-56e0-443f-9c2d-0c2642fdea77-kube-api-access-6rztk\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:54.240747 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.240656 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/964c8dfc-56e0-443f-9c2d-0c2642fdea77-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:54.240747 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.240671 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/964c8dfc-56e0-443f-9c2d-0c2642fdea77-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:31:54.447642 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.447612 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j"] Aug 02 10:31:54.456517 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.456484 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0c6483/rate-test-model-0c6483-kserve-5d78ccfcc9-k5k6j"] Aug 02 10:31:54.978702 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.978668 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="964c8dfc-56e0-443f-9c2d-0c2642fdea77" path="/var/lib/kubelet/pods/964c8dfc-56e0-443f-9c2d-0c2642fdea77/volumes" Aug 02 10:31:54.979088 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:31:54.979073 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c63a47e6-0856-4a00-b308-c6ef67168d37" path="/var/lib/kubelet/pods/c63a47e6-0856-4a00-b308-c6ef67168d37/volumes" Aug 02 10:32:05.990347 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:05.990316 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:32:27.331356 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.331321 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6"] Aug 02 10:32:27.331872 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.331820 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c63a47e6-0856-4a00-b308-c6ef67168d37" containerName="main" Aug 02 10:32:27.331872 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.331835 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c63a47e6-0856-4a00-b308-c6ef67168d37" containerName="main" Aug 02 10:32:27.331872 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.331857 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="964c8dfc-56e0-443f-9c2d-0c2642fdea77" containerName="main" Aug 02 10:32:27.331872 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.331863 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="964c8dfc-56e0-443f-9c2d-0c2642fdea77" containerName="main" Aug 02 10:32:27.332011 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.331923 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="964c8dfc-56e0-443f-9c2d-0c2642fdea77" containerName="main" Aug 02 10:32:27.332011 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.331932 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c63a47e6-0856-4a00-b308-c6ef67168d37" containerName="main" Aug 02 10:32:27.334826 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.334807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.337014 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.336995 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c167c0-openshift-default-dockercfg-2qb9x\"" Aug 02 10:32:27.346267 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.346240 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6"] Aug 02 10:32:27.435456 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.435422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-podinfo\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.435456 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.435503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e89207-737a-4efe-93e3-3ec9b15ab08c-istiod-ca-cert\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.435751 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.435571 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-certs\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.435751 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.435610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkt7r\" (UniqueName: \"kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-kube-api-access-rkt7r\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.435751 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.435664 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-socket\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.435751 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.435694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-token\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.435751 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.435731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-envoy\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.436028 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.435804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-data\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.436028 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.435853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-credential-socket\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.536871 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.536816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-podinfo\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537050 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.536890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e89207-737a-4efe-93e3-3ec9b15ab08c-istiod-ca-cert\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537050 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.536910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-certs\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537050 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.536928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkt7r\" (UniqueName: \"kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-kube-api-access-rkt7r\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537050 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.536979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-socket\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537050 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.537008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-token\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537050 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.537044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-envoy\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537364 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.537076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-data\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537364 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.537111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-credential-socket\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537488 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.537426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-certs\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537553 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.537488 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-socket\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537612 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.537558 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-credential-socket\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537701 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.537676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-data\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.537771 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.537752 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e89207-737a-4efe-93e3-3ec9b15ab08c-istiod-ca-cert\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.539247 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.539221 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-envoy\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.539378 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.539361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-podinfo\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.545450 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.545424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-token\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.545799 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.545775 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkt7r\" (UniqueName: \"kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-kube-api-access-rkt7r\") pod \"e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.644882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.644802 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:27.800427 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.800303 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6"] Aug 02 10:32:27.803184 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:32:27.803148 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e89207_737a_4efe_93e3_3ec9b15ab08c.slice/crio-5012ab31a21404e14eaff2c063d5b3d4cc6d089f1f5f36db719f1f01612948cf WatchSource:0}: Error finding container 5012ab31a21404e14eaff2c063d5b3d4cc6d089f1f5f36db719f1f01612948cf: Status 404 returned error can't find the container with id 5012ab31a21404e14eaff2c063d5b3d4cc6d089f1f5f36db719f1f01612948cf Aug 02 10:32:27.805476 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.805427 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:32:27.805593 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.805511 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:32:27.805593 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:27.805541 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:32:28.296918 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:28.296882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" event={"ID":"79e89207-737a-4efe-93e3-3ec9b15ab08c","Type":"ContainerStarted","Data":"8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5"} Aug 02 10:32:28.297086 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:28.296937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" event={"ID":"79e89207-737a-4efe-93e3-3ec9b15ab08c","Type":"ContainerStarted","Data":"5012ab31a21404e14eaff2c063d5b3d4cc6d089f1f5f36db719f1f01612948cf"} Aug 02 10:32:28.321251 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:28.321194 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" podStartSLOduration=1.321175124 podStartE2EDuration="1.321175124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:32:28.320311464 +0000 UTC m=+3569.761201950" watchObservedRunningTime="2026-08-02 10:32:28.321175124 +0000 UTC m=+3569.762065581" Aug 02 10:32:28.645431 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:28.645361 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:28.650150 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:28.650127 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:29.301238 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:29.301160 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:29.302075 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:29.302050 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:35.702667 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:35.702632 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:32:45.031962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:45.031889 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:58246->10.134.0.55:15021: read: connection reset by peer" start-of-body= Aug 02 10:32:45.032369 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:45.031980 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:58246->10.134.0.55:15021: read: connection reset by peer" Aug 02 10:32:45.395862 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:45.395775 2564 generic.go:356] "Generic (PLEG): container finished" podID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerID="8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5" exitCode=0 Aug 02 10:32:45.396015 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:45.395847 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" event={"ID":"79e89207-737a-4efe-93e3-3ec9b15ab08c","Type":"ContainerDied","Data":"8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5"} Aug 02 10:32:45.396277 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:45.396264 2564 scope.go:117] "RemoveContainer" containerID="8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5" Aug 02 10:32:45.397323 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:45.397295 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:32:45.397389 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:45.397363 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:32:45.397430 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:45.397397 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:32:46.402754 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:46.402720 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" event={"ID":"79e89207-737a-4efe-93e3-3ec9b15ab08c","Type":"ContainerStarted","Data":"3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e"} Aug 02 10:32:46.649921 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:46.649881 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:46.650115 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:46.650022 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Aug 02 10:32:46.650115 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:46.650066 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Aug 02 10:32:47.646641 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:47.646602 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Aug 02 10:32:47.647154 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:47.646676 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Aug 02 10:32:48.645799 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:48.645764 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Aug 02 10:32:48.646080 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:48.646050 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Aug 02 10:32:49.645422 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:49.645386 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Aug 02 10:32:49.646008 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:49.645977 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Aug 02 10:32:50.646138 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:50.646102 2564 patch_prober.go:28] interesting pod/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Aug 02 10:32:50.646727 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:50.646693 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Aug 02 10:32:51.649311 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:51.649237 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:51.649736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:51.649496 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:32:51.650229 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:32:51.650211 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:33:10.898981 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.898949 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6"] Aug 02 10:33:10.902800 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.902782 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:10.906487 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.906439 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5c4717\"/\"openshift-service-ca.crt\"" Aug 02 10:33:10.906487 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.906474 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5c4717\"/\"ipp-route-5c4717-kserve-self-signed-certs\"" Aug 02 10:33:10.906664 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.906438 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5c4717\"/\"default-dockercfg-cgfm7\"" Aug 02 10:33:10.906664 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.906507 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5c4717\"/\"kube-root-ca.crt\"" Aug 02 10:33:10.912549 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.912528 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6"] Aug 02 10:33:10.947242 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.947213 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-home\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:10.947343 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.947255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tls-certs\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:10.947343 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.947292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-model-cache\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:10.947343 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.947317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-dshm\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:10.947497 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.947360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlk9\" (UniqueName: \"kubernetes.io/projected/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-kube-api-access-qnlk9\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:10.947497 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:10.947410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tmp-dir\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.048651 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.048610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tmp-dir\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.048794 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.048714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-home\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.048794 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.048760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tls-certs\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.048909 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.048804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-model-cache\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.048909 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.048832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-dshm\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.048909 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.048902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlk9\" (UniqueName: \"kubernetes.io/projected/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-kube-api-access-qnlk9\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.049070 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.049043 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tmp-dir\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.049151 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.049130 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-home\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.049151 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.049142 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-model-cache\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.051062 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.051038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-dshm\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.051261 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.051242 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tls-certs\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.056573 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.056542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlk9\" (UniqueName: \"kubernetes.io/projected/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-kube-api-access-qnlk9\") pod \"ipp-route-5c4717-kserve-7fc687858c-dhbv6\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.213440 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.213407 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:11.351473 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.351430 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6"] Aug 02 10:33:11.354653 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:33:11.354628 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf07e8eb_b2c3_4f64_a5ee_a86e1a280923.slice/crio-3945b7e1604556faa48ff6c83275466f5b4e152b0997219f8986e2304758e82f WatchSource:0}: Error finding container 3945b7e1604556faa48ff6c83275466f5b4e152b0997219f8986e2304758e82f: Status 404 returned error can't find the container with id 3945b7e1604556faa48ff6c83275466f5b4e152b0997219f8986e2304758e82f Aug 02 10:33:11.529901 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.529801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" event={"ID":"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923","Type":"ContainerStarted","Data":"32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6"} Aug 02 10:33:11.529901 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.529849 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" event={"ID":"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923","Type":"ContainerStarted","Data":"3945b7e1604556faa48ff6c83275466f5b4e152b0997219f8986e2304758e82f"} Aug 02 10:33:11.547779 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:11.547727 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" podStartSLOduration=1.5477087680000001 podStartE2EDuration="1.547708768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:33:11.547109521 +0000 UTC m=+3612.987999974" watchObservedRunningTime="2026-08-02 10:33:11.547708768 +0000 UTC m=+3612.988599225" Aug 02 10:33:13.696484 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:13.696426 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:33:21.214473 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:21.214438 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:21.214874 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:21.214505 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:21.227073 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:21.227045 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:21.592662 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:21.592569 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:33:36.195267 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:36.195229 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:33:48.067966 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.067934 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht"] Aug 02 10:33:48.071519 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.071500 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.076215 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.076197 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-80d516-openshift-default-dockercfg-x5zwq\"" Aug 02 10:33:48.086763 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.086739 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht"] Aug 02 10:33:48.185309 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.185277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-istio-token\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.185511 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.185328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-certs\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.185511 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.185408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-credential-socket\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.185511 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.185444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-data\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.185511 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.185482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5deeee47-2234-4be0-8791-580e2670c08c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.185679 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.185525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5deeee47-2234-4be0-8791-580e2670c08c-istio-podinfo\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.185679 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.185609 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-socket\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.185679 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.185661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-envoy\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.185779 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.185683 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc9w8\" (UniqueName: \"kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-kube-api-access-cc9w8\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.286743 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.286715 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-certs\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.286898 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.286760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-credential-socket\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.286898 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.286876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-data\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.286978 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.286914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5deeee47-2234-4be0-8791-580e2670c08c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.287067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.287041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5deeee47-2234-4be0-8791-580e2670c08c-istio-podinfo\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.287143 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.287127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-socket\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.287205 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.287146 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-credential-socket\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.287257 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.287206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-envoy\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.287257 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.287237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9w8\" (UniqueName: \"kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-kube-api-access-cc9w8\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.287257 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.287239 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-certs\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.287410 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.287280 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-data\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.287410 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.287285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-istio-token\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.287558 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.287497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-socket\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.287682 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.287664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5deeee47-2234-4be0-8791-580e2670c08c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.289306 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.289286 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-envoy\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.289645 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.289626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5deeee47-2234-4be0-8791-580e2670c08c-istio-podinfo\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.295723 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.295689 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-istio-token\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.295831 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.295784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9w8\" (UniqueName: \"kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-kube-api-access-cc9w8\") pod \"e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.382184 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.382125 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:48.515823 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.515796 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht"] Aug 02 10:33:48.518026 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:33:48.517992 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5deeee47_2234_4be0_8791_580e2670c08c.slice/crio-5f77194d533e2411fbaed95f03d72d959cb1e8b277891161e288458accf009c2 WatchSource:0}: Error finding container 5f77194d533e2411fbaed95f03d72d959cb1e8b277891161e288458accf009c2: Status 404 returned error can't find the container with id 5f77194d533e2411fbaed95f03d72d959cb1e8b277891161e288458accf009c2 Aug 02 10:33:48.520178 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.520141 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:33:48.520308 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.520209 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:33:48.520308 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.520238 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:33:48.717624 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.717580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" event={"ID":"5deeee47-2234-4be0-8791-580e2670c08c","Type":"ContainerStarted","Data":"eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603"} Aug 02 10:33:48.717819 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.717646 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" event={"ID":"5deeee47-2234-4be0-8791-580e2670c08c","Type":"ContainerStarted","Data":"5f77194d533e2411fbaed95f03d72d959cb1e8b277891161e288458accf009c2"} Aug 02 10:33:48.739299 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:48.739243 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" podStartSLOduration=0.739220146 podStartE2EDuration="739.220146ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-02 10:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:33:48.736809867 +0000 UTC m=+3650.177700334" watchObservedRunningTime="2026-08-02 10:33:48.739220146 +0000 UTC m=+3650.180110602" Aug 02 10:33:49.383113 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:49.383080 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:49.387587 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:49.387561 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:49.722536 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:49.722502 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:49.723345 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:49.723324 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:33:57.195186 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:33:57.195150 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:34:05.764615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:05.764579 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 02 10:34:05.764996 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:05.764648 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 02 10:34:07.814924 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:07.814892 2564 generic.go:356] "Generic (PLEG): container finished" podID="5deeee47-2234-4be0-8791-580e2670c08c" containerID="eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603" exitCode=0 Aug 02 10:34:07.815354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:07.814965 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" event={"ID":"5deeee47-2234-4be0-8791-580e2670c08c","Type":"ContainerDied","Data":"eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603"} Aug 02 10:34:07.815401 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:07.815368 2564 scope.go:117] "RemoveContainer" containerID="eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603" Aug 02 10:34:07.816424 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:07.816393 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:34:07.816551 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:07.816484 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:34:07.816551 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:07.816519 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Aug 02 10:34:08.822289 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:08.822253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" event={"ID":"5deeee47-2234-4be0-8791-580e2670c08c","Type":"ContainerStarted","Data":"b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c"} Aug 02 10:34:09.382934 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:09.382896 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:34:09.383120 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:09.383094 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Aug 02 10:34:09.383166 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:09.383148 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Aug 02 10:34:10.383548 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:10.383507 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Aug 02 10:34:10.383955 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:10.383568 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Aug 02 10:34:11.383220 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:11.383178 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Aug 02 10:34:11.383425 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:11.383242 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Aug 02 10:34:12.382768 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:12.382733 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Aug 02 10:34:12.383158 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:12.382790 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Aug 02 10:34:13.383266 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:13.383226 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Aug 02 10:34:13.383766 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:13.383291 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Aug 02 10:34:14.385776 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:14.385747 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:34:14.386218 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:14.386124 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:34:14.386793 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:14.386772 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:34:25.496593 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:25.496513 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:34:29.043338 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:29.043307 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht"] Aug 02 10:34:29.043743 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:29.043540 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" containerID="cri-o://b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c" gracePeriod=30 Aug 02 10:34:29.388912 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:29.388823 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 02 10:34:29.388912 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:29.388891 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 02 10:34:34.294816 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.294792 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:34:34.338937 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.338860 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-data\") pod \"5deeee47-2234-4be0-8791-580e2670c08c\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " Aug 02 10:34:34.338937 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.338922 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5deeee47-2234-4be0-8791-580e2670c08c-istio-podinfo\") pod \"5deeee47-2234-4be0-8791-580e2670c08c\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " Aug 02 10:34:34.339132 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.338964 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5deeee47-2234-4be0-8791-580e2670c08c-istiod-ca-cert\") pod \"5deeee47-2234-4be0-8791-580e2670c08c\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " Aug 02 10:34:34.339132 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339006 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-istio-token\") pod \"5deeee47-2234-4be0-8791-580e2670c08c\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " Aug 02 10:34:34.339132 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339035 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-socket\") pod \"5deeee47-2234-4be0-8791-580e2670c08c\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " Aug 02 10:34:34.339132 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339075 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-certs\") pod \"5deeee47-2234-4be0-8791-580e2670c08c\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " Aug 02 10:34:34.339132 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339112 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-envoy\") pod \"5deeee47-2234-4be0-8791-580e2670c08c\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " Aug 02 10:34:34.339394 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339169 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-credential-socket\") pod \"5deeee47-2234-4be0-8791-580e2670c08c\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " Aug 02 10:34:34.339394 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339167 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-data" (OuterVolumeSpecName: "istio-data") pod "5deeee47-2234-4be0-8791-580e2670c08c" (UID: "5deeee47-2234-4be0-8791-580e2670c08c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:34:34.339394 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339219 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc9w8\" (UniqueName: \"kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-kube-api-access-cc9w8\") pod \"5deeee47-2234-4be0-8791-580e2670c08c\" (UID: \"5deeee47-2234-4be0-8791-580e2670c08c\") " Aug 02 10:34:34.339394 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339292 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5deeee47-2234-4be0-8791-580e2670c08c" (UID: "5deeee47-2234-4be0-8791-580e2670c08c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:34:34.339636 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339430 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5deeee47-2234-4be0-8791-580e2670c08c" (UID: "5deeee47-2234-4be0-8791-580e2670c08c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:34:34.339636 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339437 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5deeee47-2234-4be0-8791-580e2670c08c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5deeee47-2234-4be0-8791-580e2670c08c" (UID: "5deeee47-2234-4be0-8791-580e2670c08c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:34:34.339636 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339605 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5deeee47-2234-4be0-8791-580e2670c08c" (UID: "5deeee47-2234-4be0-8791-580e2670c08c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:34:34.339794 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339676 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:34:34.339794 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339694 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5deeee47-2234-4be0-8791-580e2670c08c-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:34:34.339794 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339706 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:34:34.339794 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339716 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:34:34.339794 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.339724 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:34:34.341456 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.341416 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5deeee47-2234-4be0-8791-580e2670c08c" (UID: "5deeee47-2234-4be0-8791-580e2670c08c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:34:34.341799 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.341772 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-istio-token" (OuterVolumeSpecName: "istio-token") pod "5deeee47-2234-4be0-8791-580e2670c08c" (UID: "5deeee47-2234-4be0-8791-580e2670c08c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:34:34.341799 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.341781 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5deeee47-2234-4be0-8791-580e2670c08c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5deeee47-2234-4be0-8791-580e2670c08c" (UID: "5deeee47-2234-4be0-8791-580e2670c08c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:34:34.341967 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.341820 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-kube-api-access-cc9w8" (OuterVolumeSpecName: "kube-api-access-cc9w8") pod "5deeee47-2234-4be0-8791-580e2670c08c" (UID: "5deeee47-2234-4be0-8791-580e2670c08c"). InnerVolumeSpecName "kube-api-access-cc9w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:34:34.440147 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.440123 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5deeee47-2234-4be0-8791-580e2670c08c-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:34:34.440147 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.440145 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc9w8\" (UniqueName: \"kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-kube-api-access-cc9w8\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:34:34.440284 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.440157 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5deeee47-2234-4be0-8791-580e2670c08c-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:34:34.440284 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.440165 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5deeee47-2234-4be0-8791-580e2670c08c-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:34:34.967829 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.967790 2564 generic.go:356] "Generic (PLEG): container finished" podID="5deeee47-2234-4be0-8791-580e2670c08c" containerID="b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c" exitCode=0 Aug 02 10:34:34.968019 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.967844 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" Aug 02 10:34:34.968019 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.967869 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" event={"ID":"5deeee47-2234-4be0-8791-580e2670c08c","Type":"ContainerDied","Data":"b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c"} Aug 02 10:34:34.968019 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.967955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht" event={"ID":"5deeee47-2234-4be0-8791-580e2670c08c","Type":"ContainerDied","Data":"5f77194d533e2411fbaed95f03d72d959cb1e8b277891161e288458accf009c2"} Aug 02 10:34:34.968019 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.967988 2564 scope.go:117] "RemoveContainer" containerID="b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c" Aug 02 10:34:34.978196 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.978178 2564 scope.go:117] "RemoveContainer" containerID="eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603" Aug 02 10:34:34.987071 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.987048 2564 scope.go:117] "RemoveContainer" containerID="b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c" Aug 02 10:34:34.987886 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:34:34.987857 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c\": container with ID starting with b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c not found: ID does not exist" containerID="b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c" Aug 02 10:34:34.988027 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.987896 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c"} err="failed to get container status \"b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c\": rpc error: code = NotFound desc = could not find container \"b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c\": container with ID starting with b410859bfb661f6d59fd164f989fae8d8aa9b626e98d7286b9a190aeb602b54c not found: ID does not exist" Aug 02 10:34:34.988027 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.987918 2564 scope.go:117] "RemoveContainer" containerID="eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603" Aug 02 10:34:34.988213 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:34:34.988196 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603\": container with ID starting with eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603 not found: ID does not exist" containerID="eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603" Aug 02 10:34:34.988256 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.988217 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603"} err="failed to get container status \"eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603\": rpc error: code = NotFound desc = could not find container \"eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603\": container with ID starting with eeec05b39627242d5ac0a7ae8c2229b0a60cf8622b06c66ac0d27ec85346b603 not found: ID does not exist" Aug 02 10:34:34.989758 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.989738 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht"] Aug 02 10:34:34.993714 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:34.993692 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-80d516-openshift-default-6dd84b78cb-58cht"] Aug 02 10:34:36.978881 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:36.978849 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5deeee47-2234-4be0-8791-580e2670c08c" path="/var/lib/kubelet/pods/5deeee47-2234-4be0-8791-580e2670c08c/volumes" Aug 02 10:34:45.987062 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:45.987034 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:34:53.688598 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:53.688566 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6"] Aug 02 10:34:54.501215 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:34:54.501181 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:35:08.708261 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:08.708201 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" podUID="cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" containerName="main" containerID="cri-o://32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6" gracePeriod=45 Aug 02 10:35:08.955719 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:08.955690 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:35:09.064213 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064124 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-dshm\") pod \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " Aug 02 10:35:09.064370 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064217 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnlk9\" (UniqueName: \"kubernetes.io/projected/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-kube-api-access-qnlk9\") pod \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " Aug 02 10:35:09.064370 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064260 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tmp-dir\") pod \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " Aug 02 10:35:09.064370 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064302 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-home\") pod \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " Aug 02 10:35:09.064370 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064344 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tls-certs\") pod \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " Aug 02 10:35:09.064619 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064406 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-model-cache\") pod \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\" (UID: \"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923\") " Aug 02 10:35:09.064619 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064580 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" (UID: "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:35:09.064619 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064597 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-home" (OuterVolumeSpecName: "home") pod "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" (UID: "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:35:09.064749 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064700 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" (UID: "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:35:09.064862 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064845 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:09.064909 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064864 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:09.064909 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.064874 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:09.066233 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.066212 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-dshm" (OuterVolumeSpecName: "dshm") pod "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" (UID: "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:35:09.066706 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.066681 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" (UID: "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 02 10:35:09.066706 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.066695 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-kube-api-access-qnlk9" (OuterVolumeSpecName: "kube-api-access-qnlk9") pod "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" (UID: "cf07e8eb-b2c3-4f64-a5ee-a86e1a280923"). InnerVolumeSpecName "kube-api-access-qnlk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:35:09.135012 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.134973 2564 generic.go:356] "Generic (PLEG): container finished" podID="cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" containerID="32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6" exitCode=0 Aug 02 10:35:09.135175 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.135027 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" Aug 02 10:35:09.135175 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.135054 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" event={"ID":"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923","Type":"ContainerDied","Data":"32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6"} Aug 02 10:35:09.135175 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.135106 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6" event={"ID":"cf07e8eb-b2c3-4f64-a5ee-a86e1a280923","Type":"ContainerDied","Data":"3945b7e1604556faa48ff6c83275466f5b4e152b0997219f8986e2304758e82f"} Aug 02 10:35:09.135175 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.135124 2564 scope.go:117] "RemoveContainer" containerID="32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6" Aug 02 10:35:09.144961 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.144941 2564 scope.go:117] "RemoveContainer" containerID="32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6" Aug 02 10:35:09.145249 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:35:09.145230 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6\": container with ID starting with 32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6 not found: ID does not exist" containerID="32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6" Aug 02 10:35:09.145314 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.145260 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6"} err="failed to get container status \"32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6\": rpc error: code = NotFound desc = could not find container \"32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6\": container with ID starting with 32bd6dc686821d45582a907a8c799b25fae4e77d1e9c0d8236331b374e9f24f6 not found: ID does not exist" Aug 02 10:35:09.156787 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.156763 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6"] Aug 02 10:35:09.160109 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.160085 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-5c4717/ipp-route-5c4717-kserve-7fc687858c-dhbv6"] Aug 02 10:35:09.165657 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.165637 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnlk9\" (UniqueName: \"kubernetes.io/projected/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-kube-api-access-qnlk9\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:09.165715 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.165659 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:09.165715 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:09.165670 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:10.979352 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:10.979319 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" path="/var/lib/kubelet/pods/cf07e8eb-b2c3-4f64-a5ee-a86e1a280923/volumes" Aug 02 10:35:49.800825 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:49.800789 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:35:54.353492 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:54.353390 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6"] Aug 02 10:35:54.353907 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:54.353635 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" containerID="cri-o://3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e" gracePeriod=30 Aug 02 10:35:59.612629 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.612605 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:35:59.725657 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.725628 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkt7r\" (UniqueName: \"kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-kube-api-access-rkt7r\") pod \"79e89207-737a-4efe-93e3-3ec9b15ab08c\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " Aug 02 10:35:59.725849 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.725680 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-token\") pod \"79e89207-737a-4efe-93e3-3ec9b15ab08c\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " Aug 02 10:35:59.725849 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.725706 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-podinfo\") pod \"79e89207-737a-4efe-93e3-3ec9b15ab08c\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " Aug 02 10:35:59.725849 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.725750 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-credential-socket\") pod \"79e89207-737a-4efe-93e3-3ec9b15ab08c\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " Aug 02 10:35:59.725849 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.725775 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-data\") pod \"79e89207-737a-4efe-93e3-3ec9b15ab08c\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " Aug 02 10:35:59.725849 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.725800 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e89207-737a-4efe-93e3-3ec9b15ab08c-istiod-ca-cert\") pod \"79e89207-737a-4efe-93e3-3ec9b15ab08c\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " Aug 02 10:35:59.726122 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.725854 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-envoy\") pod \"79e89207-737a-4efe-93e3-3ec9b15ab08c\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " Aug 02 10:35:59.726122 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.725886 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-certs\") pod \"79e89207-737a-4efe-93e3-3ec9b15ab08c\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " Aug 02 10:35:59.726122 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.725941 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-socket\") pod \"79e89207-737a-4efe-93e3-3ec9b15ab08c\" (UID: \"79e89207-737a-4efe-93e3-3ec9b15ab08c\") " Aug 02 10:35:59.726290 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.726113 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79e89207-737a-4efe-93e3-3ec9b15ab08c" (UID: "79e89207-737a-4efe-93e3-3ec9b15ab08c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:35:59.726349 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.726282 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e89207-737a-4efe-93e3-3ec9b15ab08c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79e89207-737a-4efe-93e3-3ec9b15ab08c" (UID: "79e89207-737a-4efe-93e3-3ec9b15ab08c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:35:59.726349 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.726314 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79e89207-737a-4efe-93e3-3ec9b15ab08c" (UID: "79e89207-737a-4efe-93e3-3ec9b15ab08c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:35:59.726457 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.726391 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:59.726457 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.726410 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e89207-737a-4efe-93e3-3ec9b15ab08c-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:59.726593 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.726505 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79e89207-737a-4efe-93e3-3ec9b15ab08c" (UID: "79e89207-737a-4efe-93e3-3ec9b15ab08c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:35:59.726593 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.726521 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-data" (OuterVolumeSpecName: "istio-data") pod "79e89207-737a-4efe-93e3-3ec9b15ab08c" (UID: "79e89207-737a-4efe-93e3-3ec9b15ab08c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:35:59.728015 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.727988 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79e89207-737a-4efe-93e3-3ec9b15ab08c" (UID: "79e89207-737a-4efe-93e3-3ec9b15ab08c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:35:59.728128 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.728018 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79e89207-737a-4efe-93e3-3ec9b15ab08c" (UID: "79e89207-737a-4efe-93e3-3ec9b15ab08c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:35:59.728128 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.728071 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-token" (OuterVolumeSpecName: "istio-token") pod "79e89207-737a-4efe-93e3-3ec9b15ab08c" (UID: "79e89207-737a-4efe-93e3-3ec9b15ab08c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:35:59.728128 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.728103 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-kube-api-access-rkt7r" (OuterVolumeSpecName: "kube-api-access-rkt7r") pod "79e89207-737a-4efe-93e3-3ec9b15ab08c" (UID: "79e89207-737a-4efe-93e3-3ec9b15ab08c"). InnerVolumeSpecName "kube-api-access-rkt7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:35:59.826987 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.826958 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:59.826987 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.826986 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:59.827160 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.826995 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:59.827160 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.827003 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:59.827160 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.827011 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:59.827160 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.827019 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e89207-737a-4efe-93e3-3ec9b15ab08c-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:35:59.827160 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:35:59.827028 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkt7r\" (UniqueName: \"kubernetes.io/projected/79e89207-737a-4efe-93e3-3ec9b15ab08c-kube-api-access-rkt7r\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:00.387030 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.386997 2564 generic.go:356] "Generic (PLEG): container finished" podID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerID="3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e" exitCode=0 Aug 02 10:36:00.387235 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.387042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" event={"ID":"79e89207-737a-4efe-93e3-3ec9b15ab08c","Type":"ContainerDied","Data":"3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e"} Aug 02 10:36:00.387235 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.387059 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" Aug 02 10:36:00.387235 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.387085 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6" event={"ID":"79e89207-737a-4efe-93e3-3ec9b15ab08c","Type":"ContainerDied","Data":"5012ab31a21404e14eaff2c063d5b3d4cc6d089f1f5f36db719f1f01612948cf"} Aug 02 10:36:00.387235 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.387107 2564 scope.go:117] "RemoveContainer" containerID="3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e" Aug 02 10:36:00.396750 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.396734 2564 scope.go:117] "RemoveContainer" containerID="8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5" Aug 02 10:36:00.404779 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.404756 2564 scope.go:117] "RemoveContainer" containerID="3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e" Aug 02 10:36:00.405016 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:36:00.404998 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e\": container with ID starting with 3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e not found: ID does not exist" containerID="3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e" Aug 02 10:36:00.405070 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.405025 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e"} err="failed to get container status \"3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e\": rpc error: code = NotFound desc = could not find container \"3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e\": container with ID starting with 3ed85a40cad2e4f5194e66c2bc0a66e0eb861e1cfe14fdf88908e627186b869e not found: ID does not exist" Aug 02 10:36:00.405070 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.405042 2564 scope.go:117] "RemoveContainer" containerID="8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5" Aug 02 10:36:00.405282 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:36:00.405264 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5\": container with ID starting with 8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5 not found: ID does not exist" containerID="8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5" Aug 02 10:36:00.405335 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.405290 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5"} err="failed to get container status \"8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5\": rpc error: code = NotFound desc = could not find container \"8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5\": container with ID starting with 8e7b1a7cbf2366bc7cd37b35f7fe2ca97577b9798774f3832a93ddd3b9e7c4c5 not found: ID does not exist" Aug 02 10:36:00.408643 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.408621 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6"] Aug 02 10:36:00.415190 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.415169 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c167c0-openshift-default-8bdc8b4-jsfz6"] Aug 02 10:36:00.978903 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:00.978863 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" path="/var/lib/kubelet/pods/79e89207-737a-4efe-93e3-3ec9b15ab08c/volumes" Aug 02 10:36:07.496520 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:07.496482 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:36:12.062597 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:12.062563 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm"] Aug 02 10:36:12.450590 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:12.450453 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" containerID="cri-o://be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188" gracePeriod=30 Aug 02 10:36:17.705859 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.705831 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:36:17.790321 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790230 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-certs\") pod \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " Aug 02 10:36:17.790321 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790318 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-credential-socket\") pod \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " Aug 02 10:36:17.790568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790346 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-envoy\") pod \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " Aug 02 10:36:17.790568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790372 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-data\") pod \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " Aug 02 10:36:17.790568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790395 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-token\") pod \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " Aug 02 10:36:17.790568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790484 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istiod-ca-cert\") pod \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " Aug 02 10:36:17.790568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790541 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "10b20eaa-3045-46a2-93c3-1ebefd00ea1d" (UID: "10b20eaa-3045-46a2-93c3-1ebefd00ea1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:36:17.790568 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790548 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-podinfo\") pod \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " Aug 02 10:36:17.790896 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790646 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwcss\" (UniqueName: \"kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-kube-api-access-dwcss\") pod \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " Aug 02 10:36:17.790896 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790678 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-socket\") pod \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\" (UID: \"10b20eaa-3045-46a2-93c3-1ebefd00ea1d\") " Aug 02 10:36:17.790896 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790691 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-data" (OuterVolumeSpecName: "istio-data") pod "10b20eaa-3045-46a2-93c3-1ebefd00ea1d" (UID: "10b20eaa-3045-46a2-93c3-1ebefd00ea1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:36:17.790896 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790704 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "10b20eaa-3045-46a2-93c3-1ebefd00ea1d" (UID: "10b20eaa-3045-46a2-93c3-1ebefd00ea1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:36:17.790896 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790773 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "10b20eaa-3045-46a2-93c3-1ebefd00ea1d" (UID: "10b20eaa-3045-46a2-93c3-1ebefd00ea1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:36:17.791157 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.790986 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "10b20eaa-3045-46a2-93c3-1ebefd00ea1d" (UID: "10b20eaa-3045-46a2-93c3-1ebefd00ea1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:36:17.791157 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.791117 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:17.791157 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.791135 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:17.791157 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.791148 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:17.791365 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.791161 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:17.791365 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.791174 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:17.792669 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.792639 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "10b20eaa-3045-46a2-93c3-1ebefd00ea1d" (UID: "10b20eaa-3045-46a2-93c3-1ebefd00ea1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:36:17.792669 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.792645 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "10b20eaa-3045-46a2-93c3-1ebefd00ea1d" (UID: "10b20eaa-3045-46a2-93c3-1ebefd00ea1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:36:17.792849 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.792676 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-token" (OuterVolumeSpecName: "istio-token") pod "10b20eaa-3045-46a2-93c3-1ebefd00ea1d" (UID: "10b20eaa-3045-46a2-93c3-1ebefd00ea1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:36:17.792849 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.792773 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-kube-api-access-dwcss" (OuterVolumeSpecName: "kube-api-access-dwcss") pod "10b20eaa-3045-46a2-93c3-1ebefd00ea1d" (UID: "10b20eaa-3045-46a2-93c3-1ebefd00ea1d"). InnerVolumeSpecName "kube-api-access-dwcss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:36:17.892308 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.892261 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:17.892308 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.892301 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:17.892308 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.892315 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:17.892565 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:17.892327 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwcss\" (UniqueName: \"kubernetes.io/projected/10b20eaa-3045-46a2-93c3-1ebefd00ea1d-kube-api-access-dwcss\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:18.482431 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.482399 2564 generic.go:356] "Generic (PLEG): container finished" podID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerID="be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188" exitCode=0 Aug 02 10:36:18.482609 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.482446 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" Aug 02 10:36:18.482609 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.482481 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" event={"ID":"10b20eaa-3045-46a2-93c3-1ebefd00ea1d","Type":"ContainerDied","Data":"be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188"} Aug 02 10:36:18.482609 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.482528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm" event={"ID":"10b20eaa-3045-46a2-93c3-1ebefd00ea1d","Type":"ContainerDied","Data":"9814b4204ff3e499f81acebf6ae7437f43ba809632fe9d659d25ecfbe8db50e7"} Aug 02 10:36:18.482609 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.482550 2564 scope.go:117] "RemoveContainer" containerID="be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188" Aug 02 10:36:18.491925 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.491911 2564 scope.go:117] "RemoveContainer" containerID="3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7" Aug 02 10:36:18.501655 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.501631 2564 scope.go:117] "RemoveContainer" containerID="be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188" Aug 02 10:36:18.502433 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:36:18.502403 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188\": container with ID starting with be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188 not found: ID does not exist" containerID="be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188" Aug 02 10:36:18.502548 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.502444 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188"} err="failed to get container status \"be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188\": rpc error: code = NotFound desc = could not find container \"be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188\": container with ID starting with be1cf365523de10373bbca6c2527c53c75b7f4bf1e9d27897a7b329cb1f8f188 not found: ID does not exist" Aug 02 10:36:18.502615 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.502557 2564 scope.go:117] "RemoveContainer" containerID="3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7" Aug 02 10:36:18.503033 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:36:18.503004 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7\": container with ID starting with 3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7 not found: ID does not exist" containerID="3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7" Aug 02 10:36:18.503114 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.503038 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7"} err="failed to get container status \"3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7\": rpc error: code = NotFound desc = could not find container \"3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7\": container with ID starting with 3dd53a7969559e6348194e88f77abd90bdb0d640b59d57ce05006a9999a35ce7 not found: ID does not exist" Aug 02 10:36:18.504741 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.504721 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm"] Aug 02 10:36:18.508504 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.508484 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e9225c-openshift-default-79dbc4bdfb-g7dxm"] Aug 02 10:36:18.986085 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:18.986049 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" path="/var/lib/kubelet/pods/10b20eaa-3045-46a2-93c3-1ebefd00ea1d/volumes" Aug 02 10:36:24.984024 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:24.983996 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-7kqkt"] Aug 02 10:36:29.836060 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:29.836020 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx"] Aug 02 10:36:29.836564 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:29.836289 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerName="istio-proxy" containerID="cri-o://b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a" gracePeriod=30 Aug 02 10:36:30.570099 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:30.570069 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65bc78c9d7-bl554_92aa30b9-77c6-4b2d-8744-b7343c15e993/authorino/0.log" Aug 02 10:36:35.082921 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.082898 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:36:35.145726 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.145653 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-socket\") pod \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " Aug 02 10:36:35.145726 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.145688 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-certs\") pod \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " Aug 02 10:36:35.145726 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.145716 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-credential-socket\") pod \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " Aug 02 10:36:35.145979 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.145749 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-data\") pod \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " Aug 02 10:36:35.145979 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.145779 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-envoy\") pod \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " Aug 02 10:36:35.145979 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.145812 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-podinfo\") pod \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " Aug 02 10:36:35.145979 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.145855 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-token\") pod \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " Aug 02 10:36:35.145979 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.145925 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" (UID: "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:36:35.146225 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.146001 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" (UID: "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:36:35.146225 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.146058 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istiod-ca-cert\") pod \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " Aug 02 10:36:35.146225 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.146088 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5whg\" (UniqueName: \"kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-kube-api-access-h5whg\") pod \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\" (UID: \"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91\") " Aug 02 10:36:35.146225 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.146119 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-data" (OuterVolumeSpecName: "istio-data") pod "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" (UID: "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:36:35.146443 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.146385 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" (UID: "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 02 10:36:35.146443 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.146437 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:35.146584 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.146444 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" (UID: "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:36:35.146584 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.146477 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:35.146584 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.146492 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:35.148073 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.148037 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-kube-api-access-h5whg" (OuterVolumeSpecName: "kube-api-access-h5whg") pod "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" (UID: "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91"). InnerVolumeSpecName "kube-api-access-h5whg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:36:35.148186 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.148041 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" (UID: "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 02 10:36:35.148186 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.148120 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-token" (OuterVolumeSpecName: "istio-token") pod "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" (UID: "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 02 10:36:35.148186 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.148128 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" (UID: "91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 02 10:36:35.246962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.246930 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:35.246962 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.246957 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:35.247143 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.246972 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:35.247143 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.246984 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:35.247143 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.246995 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5whg\" (UniqueName: \"kubernetes.io/projected/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-kube-api-access-h5whg\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:35.247143 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.247007 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Aug 02 10:36:35.569063 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.569028 2564 generic.go:356] "Generic (PLEG): container finished" podID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerID="b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a" exitCode=0 Aug 02 10:36:35.569242 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.569081 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" Aug 02 10:36:35.569242 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.569110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" event={"ID":"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91","Type":"ContainerDied","Data":"b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a"} Aug 02 10:36:35.569242 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.569164 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx" event={"ID":"91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91","Type":"ContainerDied","Data":"71aebed1f7b00e53786e906697e3d59947a046cf0b12dd525b2de58744640427"} Aug 02 10:36:35.569242 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.569186 2564 scope.go:117] "RemoveContainer" containerID="b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a" Aug 02 10:36:35.578891 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.578867 2564 scope.go:117] "RemoveContainer" containerID="abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7" Aug 02 10:36:35.586952 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.586937 2564 scope.go:117] "RemoveContainer" containerID="b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a" Aug 02 10:36:35.587199 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:36:35.587180 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a\": container with ID starting with b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a not found: ID does not exist" containerID="b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a" Aug 02 10:36:35.587264 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.587207 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a"} err="failed to get container status \"b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a\": rpc error: code = NotFound desc = could not find container \"b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a\": container with ID starting with b68f0d77d5814596925fd3627482b2d2a90f48f5fecd5ad065f2ceb01c9d121a not found: ID does not exist" Aug 02 10:36:35.587264 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.587224 2564 scope.go:117] "RemoveContainer" containerID="abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7" Aug 02 10:36:35.587450 ip-10-0-130-182 kubenswrapper[2564]: E0802 10:36:35.587430 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7\": container with ID starting with abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7 not found: ID does not exist" containerID="abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7" Aug 02 10:36:35.587502 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.587456 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7"} err="failed to get container status \"abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7\": rpc error: code = NotFound desc = could not find container \"abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7\": container with ID starting with abe5f9103a14c127a85a3ecde1d7e958f018f5c9622046345639112f9f3a9dd7 not found: ID does not exist" Aug 02 10:36:35.592134 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.592110 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx"] Aug 02 10:36:35.596117 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:35.596095 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-0c6483-openshift-default-7c948bff9b-z49xx"] Aug 02 10:36:36.815549 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:36.815525 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5694fb54d4-trvzp_a144164b-d9cd-42d5-9c29-329bae2aa0ef/copy-manifests/0.log" Aug 02 10:36:36.823117 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:36.823093 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5694fb54d4-trvzp_a144164b-d9cd-42d5-9c29-329bae2aa0ef/manager/0.log" Aug 02 10:36:36.977976 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:36.977950 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" path="/var/lib/kubelet/pods/91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91/volumes" Aug 02 10:36:38.198598 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:38.198564 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65bc78c9d7-bl554_92aa30b9-77c6-4b2d-8744-b7343c15e993/authorino/0.log" Aug 02 10:36:38.432041 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:38.432013 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-lcbtj_3f02359d-42e9-41e0-8b82-80959b07b999/manager/0.log" Aug 02 10:36:38.545138 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:38.545114 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-nqhlp_189dbbf3-6076-40cb-8422-8259461886e3/kuadrant-console-plugin/0.log" Aug 02 10:36:38.787256 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:38.787227 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-7kqkt_4047abd3-73ff-4253-8362-434614d61353/limitador/0.log" Aug 02 10:36:39.511594 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:39.511568 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7569d6b956-9hnwz_c9ad7691-ec0a-483c-af2e-1828652793b3/kube-auth-proxy/0.log" Aug 02 10:36:40.917401 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:40.917364 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-wq4ph_870b531d-c265-4263-be18-ffc2a6d618f8/main/0.log" Aug 02 10:36:49.600044 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:49.600012 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qjr5v_27a5cbb8-9293-4a54-9016-a822e9633ed2/global-pull-secret-syncer/0.log" Aug 02 10:36:49.754755 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:49.754724 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n4td2_3d42b000-3950-4ca8-b754-e38a28f50662/konnectivity-agent/0.log" Aug 02 10:36:49.780281 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:49.780249 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-182.ec2.internal_9cf72a7270824af7232dadef49144540/haproxy/0.log" Aug 02 10:36:55.072825 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:55.072791 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65bc78c9d7-bl554_92aa30b9-77c6-4b2d-8744-b7343c15e993/authorino/0.log" Aug 02 10:36:55.144183 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:55.144131 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-lcbtj_3f02359d-42e9-41e0-8b82-80959b07b999/manager/0.log" Aug 02 10:36:55.172678 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:55.172653 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-nqhlp_189dbbf3-6076-40cb-8422-8259461886e3/kuadrant-console-plugin/0.log" Aug 02 10:36:55.361777 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:55.361688 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-7kqkt_4047abd3-73ff-4253-8362-434614d61353/limitador/0.log" Aug 02 10:36:57.017245 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.017212 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47233862-95a6-47dd-b064-4385f0aca30b/alertmanager/0.log" Aug 02 10:36:57.066650 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.066626 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47233862-95a6-47dd-b064-4385f0aca30b/config-reloader/0.log" Aug 02 10:36:57.096442 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.096416 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47233862-95a6-47dd-b064-4385f0aca30b/kube-rbac-proxy-web/0.log" Aug 02 10:36:57.121758 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.121731 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47233862-95a6-47dd-b064-4385f0aca30b/kube-rbac-proxy/0.log" Aug 02 10:36:57.149759 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.149736 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47233862-95a6-47dd-b064-4385f0aca30b/kube-rbac-proxy-metric/0.log" Aug 02 10:36:57.174964 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.174940 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47233862-95a6-47dd-b064-4385f0aca30b/prom-label-proxy/0.log" Aug 02 10:36:57.196774 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.196752 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_47233862-95a6-47dd-b064-4385f0aca30b/init-config-reloader/0.log" Aug 02 10:36:57.279389 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.279360 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-fv4ph_073c08eb-a4a8-490f-8af8-6453451cf4dd/kube-state-metrics/0.log" Aug 02 10:36:57.296704 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.296680 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-fv4ph_073c08eb-a4a8-490f-8af8-6453451cf4dd/kube-rbac-proxy-main/0.log" Aug 02 10:36:57.320309 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.320285 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-fv4ph_073c08eb-a4a8-490f-8af8-6453451cf4dd/kube-rbac-proxy-self/0.log" Aug 02 10:36:57.349589 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.349516 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f75c78d86-6wzwh_c2d61a54-8add-43e7-bc0d-f6752f8444f7/metrics-server/0.log" Aug 02 10:36:57.376347 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.376313 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-w95wf_ecdee2fd-65c3-4b43-abe8-ddd5b7b46af7/monitoring-plugin/0.log" Aug 02 10:36:57.412408 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.412375 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kpkq_47487674-5e42-4214-a3e9-59ebf2d12c7b/node-exporter/0.log" Aug 02 10:36:57.435046 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.435015 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kpkq_47487674-5e42-4214-a3e9-59ebf2d12c7b/kube-rbac-proxy/0.log" Aug 02 10:36:57.463208 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.463187 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5kpkq_47487674-5e42-4214-a3e9-59ebf2d12c7b/init-textfile/0.log" Aug 02 10:36:57.671789 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.671757 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-dzcm5_2efa155f-3829-42ea-8166-e046d9e61392/kube-rbac-proxy-main/0.log" Aug 02 10:36:57.694676 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.694649 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-dzcm5_2efa155f-3829-42ea-8166-e046d9e61392/kube-rbac-proxy-self/0.log" Aug 02 10:36:57.716992 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.716962 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-dzcm5_2efa155f-3829-42ea-8166-e046d9e61392/openshift-state-metrics/0.log" Aug 02 10:36:57.754301 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.754270 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13dc4b27-5c2e-4a0b-88bb-47dcc0746435/prometheus/0.log" Aug 02 10:36:57.775397 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.775359 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13dc4b27-5c2e-4a0b-88bb-47dcc0746435/config-reloader/0.log" Aug 02 10:36:57.797676 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.797648 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13dc4b27-5c2e-4a0b-88bb-47dcc0746435/thanos-sidecar/0.log" Aug 02 10:36:57.821377 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.821356 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13dc4b27-5c2e-4a0b-88bb-47dcc0746435/kube-rbac-proxy-web/0.log" Aug 02 10:36:57.843699 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.843679 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13dc4b27-5c2e-4a0b-88bb-47dcc0746435/kube-rbac-proxy/0.log" Aug 02 10:36:57.867335 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.867317 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13dc4b27-5c2e-4a0b-88bb-47dcc0746435/kube-rbac-proxy-thanos/0.log" Aug 02 10:36:57.889031 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.889013 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_13dc4b27-5c2e-4a0b-88bb-47dcc0746435/init-config-reloader/0.log" Aug 02 10:36:57.925819 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.925749 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-8fvdr_a4ff2e2c-ba72-49e2-b161-e11788ee3560/prometheus-operator/0.log" Aug 02 10:36:57.948512 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.948489 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-8fvdr_a4ff2e2c-ba72-49e2-b161-e11788ee3560/kube-rbac-proxy/0.log" Aug 02 10:36:57.985190 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:57.985169 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-ffr77_abe2537e-513b-4ae4-9ae0-8632d800488a/prometheus-operator-admission-webhook/0.log" Aug 02 10:36:58.014160 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:58.014140 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fd7c86d-cgsd6_cfd25b47-3ce0-48ea-a491-fd08ebb6a523/telemeter-client/0.log" Aug 02 10:36:58.035302 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:58.035273 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fd7c86d-cgsd6_cfd25b47-3ce0-48ea-a491-fd08ebb6a523/reload/0.log" Aug 02 10:36:58.058778 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:58.058757 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fd7c86d-cgsd6_cfd25b47-3ce0-48ea-a491-fd08ebb6a523/kube-rbac-proxy/0.log" Aug 02 10:36:58.093681 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:58.093660 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/thanos-query/0.log" Aug 02 10:36:58.115235 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:58.115220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/kube-rbac-proxy-web/0.log" Aug 02 10:36:58.137269 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:58.137252 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/kube-rbac-proxy/0.log" Aug 02 10:36:58.159354 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:58.159333 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/prom-label-proxy/0.log" Aug 02 10:36:58.181564 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:58.181485 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/kube-rbac-proxy-rules/0.log" Aug 02 10:36:58.202116 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:58.202098 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b59bc865b-6r822_315e54db-6885-4ae2-83a2-bed703db3b9d/kube-rbac-proxy-metrics/0.log" Aug 02 10:36:59.814302 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814225 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9"] Aug 02 10:36:59.814660 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814636 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" Aug 02 10:36:59.814660 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814649 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" Aug 02 10:36:59.814736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814664 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" Aug 02 10:36:59.814736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814669 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" Aug 02 10:36:59.814736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814677 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerName="istio-proxy" Aug 02 10:36:59.814736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814682 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerName="istio-proxy" Aug 02 10:36:59.814736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814693 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" containerName="main" Aug 02 10:36:59.814736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814702 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" containerName="main" Aug 02 10:36:59.814736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814708 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" Aug 02 10:36:59.814736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814713 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" Aug 02 10:36:59.814736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814723 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" Aug 02 10:36:59.814736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814727 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" Aug 02 10:36:59.814736 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814737 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814743 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814753 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814758 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814772 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerName="istio-proxy" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814781 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerName="istio-proxy" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814838 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="91fb8b11-2ba1-4cb0-a80e-d6f240f8cb91" containerName="istio-proxy" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814848 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf07e8eb-b2c3-4f64-a5ee-a86e1a280923" containerName="main" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814858 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="10b20eaa-3045-46a2-93c3-1ebefd00ea1d" containerName="istio-proxy" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814864 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814870 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e89207-737a-4efe-93e3-3ec9b15ab08c" containerName="istio-proxy" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814875 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" Aug 02 10:36:59.815067 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.814890 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5deeee47-2234-4be0-8791-580e2670c08c" containerName="istio-proxy" Aug 02 10:36:59.819239 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.819221 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:36:59.821355 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.821331 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4kwt9\"/\"openshift-service-ca.crt\"" Aug 02 10:36:59.822173 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.822154 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4kwt9\"/\"default-dockercfg-97nlp\"" Aug 02 10:36:59.822235 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.822171 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4kwt9\"/\"kube-root-ca.crt\"" Aug 02 10:36:59.825356 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.825336 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9"] Aug 02 10:36:59.959737 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.959703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-podres\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:36:59.959882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.959752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp5bp\" (UniqueName: \"kubernetes.io/projected/5610fc1f-36e3-4499-935f-ffc4cba5309c-kube-api-access-vp5bp\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:36:59.959882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.959779 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-proc\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:36:59.959882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.959846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-sys\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:36:59.959882 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:36:59.959878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-lib-modules\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.060430 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.060401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-podres\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.060651 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.060495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5bp\" (UniqueName: \"kubernetes.io/projected/5610fc1f-36e3-4499-935f-ffc4cba5309c-kube-api-access-vp5bp\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.060651 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.060526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-proc\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.060651 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.060579 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-podres\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.060651 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.060587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-sys\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.060651 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.060637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-sys\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.060651 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.060650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-lib-modules\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.060927 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.060636 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-proc\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.060927 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.060757 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5610fc1f-36e3-4499-935f-ffc4cba5309c-lib-modules\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.068154 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.068093 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5bp\" (UniqueName: \"kubernetes.io/projected/5610fc1f-36e3-4499-935f-ffc4cba5309c-kube-api-access-vp5bp\") pod \"perf-node-gather-daemonset-8vnd9\" (UID: \"5610fc1f-36e3-4499-935f-ffc4cba5309c\") " pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.130402 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.130367 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.261569 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.261516 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9"] Aug 02 10:37:00.263866 ip-10-0-130-182 kubenswrapper[2564]: W0802 10:37:00.263842 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5610fc1f_36e3_4499_935f_ffc4cba5309c.slice/crio-f521f851c365b12b74f6668de9a7767e991aa62a0f39119d98ae3c5ceed92bb4 WatchSource:0}: Error finding container f521f851c365b12b74f6668de9a7767e991aa62a0f39119d98ae3c5ceed92bb4: Status 404 returned error can't find the container with id f521f851c365b12b74f6668de9a7767e991aa62a0f39119d98ae3c5ceed92bb4 Aug 02 10:37:00.690678 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.690594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" event={"ID":"5610fc1f-36e3-4499-935f-ffc4cba5309c","Type":"ContainerStarted","Data":"432c748f966b60d3e3023373d9cb81772b89759fec905a8d28bb0b6dd84f6666"} Aug 02 10:37:00.690678 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.690649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" event={"ID":"5610fc1f-36e3-4499-935f-ffc4cba5309c","Type":"ContainerStarted","Data":"f521f851c365b12b74f6668de9a7767e991aa62a0f39119d98ae3c5ceed92bb4"} Aug 02 10:37:00.690678 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.690674 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:00.708378 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:00.708329 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" podStartSLOduration=1.708313376 podStartE2EDuration="1.708313376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-02 10:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-02 10:37:00.706557584 +0000 UTC m=+3842.147448041" watchObservedRunningTime="2026-08-02 10:37:00.708313376 +0000 UTC m=+3842.149203832" Aug 02 10:37:02.514252 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:02.514225 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mdqdc_f633b7b6-33dd-492f-8cd0-1a6604102b82/dns/0.log" Aug 02 10:37:02.547315 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:02.547285 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mdqdc_f633b7b6-33dd-492f-8cd0-1a6604102b82/kube-rbac-proxy/0.log" Aug 02 10:37:02.627852 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:02.627829 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dbgqp_3310e94b-d9e4-4f11-8c91-b54387f17a5e/dns-node-resolver/0.log" Aug 02 10:37:03.185501 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:03.185448 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c7cbb5c7f-q759r_1bff3366-0670-4bc9-8327-a326370b731a/registry/0.log" Aug 02 10:37:03.261942 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:03.261911 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jxtkm_6be5c331-cc91-4b21-b720-fbf73f0da999/node-ca/0.log" Aug 02 10:37:04.592640 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:04.592610 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7569d6b956-9hnwz_c9ad7691-ec0a-483c-af2e-1828652793b3/kube-auth-proxy/0.log" Aug 02 10:37:05.702038 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:05.702004 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sxvlb_9028085f-6881-4a18-92b3-343810822d8d/serve-healthcheck-canary/0.log" Aug 02 10:37:06.403139 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:06.403108 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2stw_5c4288d5-ac54-473a-88b4-ce6df5d95b96/kube-rbac-proxy/0.log" Aug 02 10:37:06.423432 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:06.423409 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2stw_5c4288d5-ac54-473a-88b4-ce6df5d95b96/exporter/0.log" Aug 02 10:37:06.447145 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:06.447106 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2stw_5c4288d5-ac54-473a-88b4-ce6df5d95b96/extractor/0.log" Aug 02 10:37:06.705571 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:06.705549 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4kwt9/perf-node-gather-daemonset-8vnd9" Aug 02 10:37:09.699796 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:09.699725 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5694fb54d4-trvzp_a144164b-d9cd-42d5-9c29-329bae2aa0ef/manager/0.log" Aug 02 10:37:09.727708 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:09.727687 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5694fb54d4-trvzp_a144164b-d9cd-42d5-9c29-329bae2aa0ef/copy-manifests/0.log" Aug 02 10:37:10.887608 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:10.887579 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7569d6b956-9hnwz_c9ad7691-ec0a-483c-af2e-1828652793b3/kube-auth-proxy/0.log" Aug 02 10:37:10.994577 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:10.994547 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-df7456b48-j4kkw_d7a7ef85-18be-4bda-95c0-2541f54850db/manager/0.log" Aug 02 10:37:12.756412 ip-10-0-130-182 kubenswrapper[2564]: I0802 10:37:12.756378 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7569d6b956-9hnwz_c9ad7691-ec0a-483c-af2e-1828652793b3/kube-auth-proxy/0.log"